Image generation apparatus and image generation method

ABSTRACT

A characteristic point extraction unit extracts a characteristic point in an overlapping region of a plurality of component images picked up using a camera built in portable equipment and having pan angles or elevation angles different from each other. A self-position estimation unit estimates three-dimensional coordinate values of the characteristic point and three-dimensional coordinate values of a camera position, which are associated with each other, at the same time in two component images in which the camera positions are displaced from each other. A panoramic image generation unit adjusts, based on the estimated three-dimensional coordinate values of the camera position, the three-dimensional coordinate values of the characteristic point of the component image to correct the component image and combine a plurality of component images having pan angles or elevation angles different from each other to generate a synthesis image.

TECHNICAL FIELD

The present invention relates to an image generation apparatus andmethod. Specifically, the present invention relates to an apparatus forand a method of generating a synthesized image.

BACKGROUND ART

A digital still camera and a digital video camera are spread, and anopportunity is increasing in which a picked up still picture or movingpicture is stored into a computer and is read or worked or is displayedon a screen of a game machine or a television (TV) system. Also it ispopular to upload a picked up moving picture to a posting site of theInternet and share the moving picture with different camera users.

Some digital cameras are capable of picking up a panoramic image, and ithas become possible to readily pick up a panoramic image having a wideview angle. Also a software tool for generating a panoramic image bysticking a plurality of images picked up by a digital camera togetherwhile the image pickup direction is successively changed is utilizedfrequently.

Also a portable apparatus such as a portable telephone set with a camerais available in which a panoramic image pickup mode or an applicationfor synthesizing a panoramic image is provided and, while theinclination of the portable apparatus is successively changed, imagescan be picked up and automatically synthesized into a panoramic image.

CITATION LIST Patent Literature

[PTL 1]

JP 2011-76249A

SUMMARY Technical Problem

However, if panoramic image pickup is carried out using a digitalcamera, portable equipment with a camera or like equipment, then anaccurate panoramic image may not be able to be picked up due to camerashake, displacement of a rotational axis or the like and image matchingmay not be carried out successfully, resulting in occurrence of an errorin a synthesized panoramic image.

The present invention has been made in view of such a subject as justdescribed, and it is an object of the present invention to provide atechnology by which a synthesis image can be generated accurately.

Solution to Problem

In order to solve the subject described above, an image generationapparatus of a mode of the present invention includes: a characteristicpoint extraction unit configured to extract a characteristic point in anoverlapping region of a plurality of component images picked up using acamera built in portable equipment and having pan angles or elevationangles different from each other; a self-position estimation unitconfigured to estimate three-dimensional coordinate values of thecharacteristic point and three-dimensional coordinate values of a cameraposition, which are associated with each other, at the same time in twocomponent images in which the camera positions are displaced from eachother; and a synthesis image generation unit configured to adjust, basedon the estimated three-dimensional coordinate values of the cameraposition, the three-dimensional coordinate values of the characteristicpoint of the component image to correct the component image and combinea plurality of component images having pan angles or elevation anglesdifferent from each other to generate a synthesis image.

Another mode of the present invention is an image generation method.This method includes: a characteristic point extraction step ofextracting a characteristic point in an overlapping region of aplurality of component images picked up using a camera built in aportable equipment and having pan angles or elevation angles differentfrom each other; a self-position estimation step of estimatingthree-dimensional coordinate values of the characteristic point andthree-dimensional coordinate values of a camera position, which areassociated with each other, at the same time in two component images inwhich the camera positions are displaced from each other; and asynthesis image generation step of adjusting, based on the estimatedthree-dimensional coordinate values of the camera position, thethree-dimensional coordinate values of the characteristic point of thecomponent image to correct the component image and combine a pluralityof component images having pan angles or elevation angles different fromeach other to generate a synthesis image.

It is to be noted that also an arbitrary combination of the componentsdescribed above and also representations of the present invention whichare transformed between a method, an apparatus, a system, a computerprogram, a data structure, a recording medium and so forth are effectiveas modes of the present invention.

Advantageous Effect of Invention

With the present invention, a synthesis image can be generatedaccurately.

BRIEF DESCRIPTION OF DRAWINGS

FIGS. 1(a) and 1(b) are appearance views of a portable terminal used forpanoramic image pickup.

FIG. 2 is a view illustrating an inclination of the portable terminal.

FIGS. 3(a) and 3(b) are views illustrating an orientation of theportable terminal.

FIGS. 4(a) to 4(d) are views illustrating an image pickup direction whena panoramic image is picked up using a rear side camera of the portableterminal.

FIGS. 5(a) and 5(b) are views illustrating an azimuth angle θ of acamera and an elevation angle φ of a camera.

FIGS. 6(a) to 6(c) are views illustrating a panoramic image picked upwhen an initial position of the rear side camera of the portableterminal is in a direction of the azimuth angle θ.

FIGS. 7(a) to 7(c) are views illustrating a panoramic image picked upwhen the elevation angle φ of the rear side camera=60 degrees.

FIG. 8A is a view illustrating a method of connecting a plurality ofimages to each other to generate a panoramic image.

FIG. 8B is a view illustrating another method of connecting a pluralityof images to each other to generate a panoramic image.

FIG. 9 is a functional block diagram of a panoramic image generationapparatus.

FIG. 10 is a flow chart illustrating a procedure of image pickup of apanoramic image.

FIG. 11 is a flow chart illustrating a detailed procedure of an imagepickup timing decision process of FIG. 10.

FIGS. 12(a) to 12(c) are views illustrating a panoramic image pickupguide by an image pickup guide portion.

FIGS. 13(a) and 13(b) are views illustrating an example of componentimages and a panoramic image.

FIG. 14 is a view illustrating a panoramic image during image pickup.

FIG. 15 is a view illustrating an example of a guide of an image pickuporder of the panoramic image pickup.

FIG. 16 is a schematic view of a see-through type head-mounted display.

FIGS. 17(a) and 17(b) are views illustrating examples of a displayscreen image of the see-through type HMD.

FIG. 18 is a functional block diagram of a panoramic image generationapparatus according to a different embodiment.

FIG. 19 is a flow chart illustrating a procedure of image pickup of apanoramic image according to the different embodiment.

FIG. 20 is a flow chart illustrating a detailed procedure of aself-position estimation process of FIG. 19.

FIGS. 21(a) and 21(b) are views depicting a control point detectedbetween two component images adjacent each other.

FIG. 22 is a view illustrating a manner in which the two adjacentcomponent images are aligned and synthesized based on the control point.

FIG. 23 is a view illustrating an example of a guide for urging toadjust a displacement of a camera position.

FIG. 24 is a functional block diagram of a panoramic image generationapparatus according to a further different embodiment.

FIG. 25 is a flow chart illustrating a procedure of image pickup of apanoramic image according to the further different embodiment.

FIGS. 26(a) to 26(c) are views illustrating an example of a format of apicked up image to which an attention region is set.

FIGS. 27(a) to 27(f) are views illustrating a method of setting anattention region on a touch panel by a user.

FIG. 28 is a view illustrating an attention region set to an α plane ora layer for exclusive use to be overlapped with a panoramic image.

FIG. 29 is a functional block diagram of a panoramic image generationapparatus according to a still further different embodiment.

FIGS. 30(a) to 30(c) are views illustrating a disposition of and aparallax between a front side camera and a rear side camera of aportable terminal.

FIG. 31(a) is a view illustrating cylindrical images picked up by therear side camera and FIG. 31(b) is a view illustrating cylindricalimages picked up by the front side camera.

FIG. 32(a) is a view illustrating a rear side panoramic image picked upby the rear side camera and FIG. 32(b) is a view illustrating a frontside panoramic image picked up by the front side camera.

FIG. 33 is a view illustrating a state in which a rear side panoramicimage picked up by the rear side camera and a front side panoramic imagepicked up by the front side camera are connected to each other.

FIG. 34 is a view illustrating an example of a front side panoramicimage picked up by the front side camera and a rear side panoramic imagepicked up by the rear side camera.

FIG. 35 is a view illustrating an example of a guide when theconsistency between a front side panoramic image and a rear sidepanoramic image is low.

FIG. 36 is a flow chart illustrating a procedure of image pickup of apanoramic image according to a yet further different embodiment.

FIG. 37 is a flow chart illustrating a detailed procedure of a panoramicimage pickup process by double-sided cameras of FIG. 36.

FIG. 38 is a view illustrating a portable terminal in which two camerashaving optical axes which are different in the position and thedirection from each other are incorporated.

FIG. 39 is a view illustrating a portable terminal in which threecameras having optical axes which are different in the position fromeach other are incorporated.

FIG. 40 is a view illustrating a portable terminal in which threecameras having optical axes which are different in the direction fromeach other are incorporated.

FIGS. 41(a) and 41(b) are views illustrating a portable terminal inwhich two cameras having natures different from each other areincorporated.

FIGS. 42(a) to 42(c) are views illustrating variations of the number ofand the direction of cameras incorporated in a portable terminal.

FIG. 43 is a view illustrating a portable terminal in which four camerasare incorporated.

FIGS. 44(a) and 44(b) are views illustrating a portable terminal inwhich a movable camera whose direction can be adjusted is incorporated.

DESCRIPTION OF EMBODIMENTS Embodiment 1

FIGS. 1(a) and 1(b) are appearance views of a portable terminal 200 usedfor panoramic image pickup. The portable terminal 200 includes cameras210 and 212 and a display 240 and can execute various types ofapplications. The portable terminal 200 may have a communicationfunction.

FIG. 1(a) depicts a front face of the portable terminal 200. The frontside (front) camera 210 and the display 240 are provided on the frontface of the portable terminal 200. A user can pick up, for example, animage of the own face using the front side camera 210 and display thepicked up image on the display 240 or transmit the picked up image as amoving picture or a still picture to a communication partner. Thedisplay 240 may include a touch panel. The user can directly touch thetouch panel with a finger to input an operation to a screen of thedisplay 240 while watching the screen image of the display 240.

FIG. 1(b) depicts a rear face of the portable terminal 200. The rearside (rear) camera 212 is provided on a rear face of the portableterminal 200. The user can pick up an image of, for example, a landscapeor a friend using the rear side camera 212 and display the picked upimage on the display 240 or transmit the picked up image as a movingpicture or a still picture to the communication partner. A touch panelmay be provided also on the rear face of the portable terminal 200.Consequently, the user can hold the portable terminal 200 and touch thetouch panel on the rear face of the portable terminal 200 with a fingerto input a predetermined operation while watching the display 240 on thefront face of the portable terminal 200.

It is to be noted that the front side camera 210 of the portableterminal 200 is an option, and only the rear side camera 212 may beprovided.

The portable terminal 200 includes at least one of a three-axis gyro(angular velocity) sensor, a three-axis acceleration sensor and athree-axis geomagnetic sensor and can detect an inclination of theportable terminal 200 with respect to three axes to acquire postureinformation. In addition, the three-axis geomagnetic sensor can detect ageomagnetism vector with respect to three axes to acquire orientationinformation. The portable terminal 200 may further include a GPS (GlobalPositioning System) receiver as an option so that it can acquireposition information from the GPS. Further, the portable terminal 200may include a transmitter-receiver for wireless communication to acquireposition information from a wireless base station.

While, as an example of portable equipment used for panoramic imagepickup, the portable terminal 200 is described here, the portableequipment used for panoramic image pickup may be a digital camera whichincludes at least one of a three-axis gyro sensor, a three-axisacceleration sensor and a three-axis geomagnetic sensor and can pick upa still picture or a moving picture. Alternatively, the portableequipment used for panoramic image pickup may be a game machine or aportable telephone set.

FIG. 2 is a view illustrating an inclination of the portable terminal200. If the display face of the portable terminal 200 is placed on an XZplane and a direction perpendicular to the display face is defined as aY axis, then rotation of the portable terminal 200 around the X axis istilting; rotation around the Y axis is rolling; and rotation around theZ axis is panning. The portable terminal 200 has incorporated therein atleast one of a three-axis gyro sensor, a three-axis acceleration sensorand a three-axis geomagnetic sensor, and an inclination of the portableterminal 200 can be detected with respect to the three axes by detectingtilting, rolling and panning of the portable terminal 200. Therefore,posture information of the portable terminal 200 when the portableterminal 200 is held by the user can be acquired. Also it is possible todetect a moving direction or a moving speed of the portable terminal 200by the gyro sensor and/or the acceleration sensor.

FIGS. 3(a) and 3(b) are views illustrating an orientation of theportable terminal 200. A three-axis geomagnetic sensor is incorporatedin the portable terminal 200, and a geomagnetism vector F havingthree-axis components can be detected as depicted in FIG. 3(a). Sincethe three-axis gyro sensor and/or the three-axis acceleration sensor areincorporated in the portable terminal 200 as described hereinabove, alsoit is possible to detect a gravity vector V.

FIG. 3(b) is a view illustrating a method of determining an orientationvector H from the geomagnetism vector F and the gravity vector V. If thegeomagnetism vector F is projected to a horizontal plane 260perpendicular to the gravity vector V, then the orientation vector H canbe obtained. The orientation vector H is a horizontal component of thegeomagnetism and provides an absolute orientation. In this manner,whichever inclination the portable terminal 200 has, an absoluteorientation of north, south, east and west can be detected using thegravity vector V detected by the three-axis gyro sensor and/orthree-axis acceleration sensor and the geomagnetism vector F detected bythe three-axis geomagnetic sensor.

Further, if a GPS sensor is incorporated in the portable terminal 200,then also it is possible to acquire such position information as alatitude and a longitude.

If posture information and orientation information of the portableterminal 200 are used, then images picked up by the camera of theportable terminal 200 while the inclination of the portable terminal 200is successively changed can be stitched (sewed up) to each other togenerate a celestial sphere panoramic image. The stitching method may besimilar to a generation method of a celestial sphere panoramic image bya pan-tilt camera. If posture information from the three-axis gyrosensor and/or three-axis acceleration sensor is used, then a pan angleand a tilt angle of the camera can be obtained. Further, if orientationinformation from the three-axis geomagnetic sensor is used, then the panangle of the camera can be associated with an absolute orientationangle. This makes it possible to carry out panoramic image pickup by thecamera of the portable terminal 200 even if such expensive equipment asa pan-tilt camera is not available.

FIGS. 4(a) to 4(d) are views illustrating an image pickup direction whenthe rear side camera 212 of the portable terminal 200 is used to pick upa panoramic image. Also it is possible to pick up a panoramic imageusing the front side camera 210 of the portable terminal 200, and asoccasion demands, also it is possible to pick up a panoramic image usingboth of the front side camera 210 and the rear side camera 212. However,in order to simplify the description, a case in which the rear sidecamera 212 is used to pick up a panoramic image is described here.

As depicted in FIG. 4(d), the user would incline the portable terminal200 freely to pick up an image by the rear side camera 212. If theportable terminal 200 is rotated around the Z axis, then the pan angleof the camera changes; if the portable terminal 200 is rotated aroundthe X axis, then the tilt angle of the camera changes; and if theportable terminal 200 is rotated around the Y axis, then the roll angleof the camera changes. Here, the Z axis is a vertical axis (axis in thedirection of gravitational force).

FIG. 4(a) is a top plan view of the portable terminal 200. An initialdirection (Y-axis direction) of the rear side camera 212 of the portableterminal 200 is defined as a pan angle of 0 degrees. The pan angle ofthe rear side camera 212 can be changed arbitrarily (as an example,within a range from −180 degrees to +180 degrees) around the Z axis.

FIG. 4(b) is a front elevational view of the portable terminal 200. Astate in which a horizontally elongated side face of the portableterminal 200 is held horizontally by the user is defined as a roll angleof 0 degrees. The roll angle of the rear side camera 212 can be changedarbitrarily (as an example, within a range from −180 degrees to +180degrees) around the Y axis.

FIG. 4(c) is a side elevational view of the portable terminal 200. Astate in which the rear face of the portable terminal 200 is heldperpendicularly to the ground is defined as a tilt angle of 0 degrees.The tilt angle of the rear side camera 212 can be changed arbitrarily(as an example, within a range from −90 degrees to +90 degrees) aroundthe X axis.

In order to provide information relating to an image pickup orientationto a panoramic image picked up by the rear side camera 212 of theportable terminal 200 of FIG. 4(d), it is necessary to record in whichorientation the rear side camera 212 of the portable terminal 200 isdirected upon image pickup. To this end, the three-axis gyro sensorand/or three-axis acceleration sensor and the three-axis geomagneticsensor incorporated in the portable terminal 200 are used.

FIG. 5(a) is a view illustrating the orientation angle θ of the rearside camera 212 of the portable terminal 200, and FIG. 5(b) is a viewillustrating the elevation angle φ of the rear side camera 212 of theportable terminal 200. FIG. 5(a) is a top plan view of the portableterminal 200, and the rear side camera 212 is directed, at an initialposition of image pickup, to a direction 220 displaced by theorientation angle θ toward the east from the true north. This anglecorresponds to the pan angle of 0 degrees. In other words, theorientation angle of the reference direction 220 of the pan angle is θ.When a panoramic image is picked up, a panoramic image of a subject ispicked up while the pan angle is successively changed, as an example,over the range from −180 degrees to +180 degrees with respect to thereference direction 220 of the orientation angle θ.

FIG. 5(b) is a side elevational view of the portable terminal 200, andthe elevation angle φ is an angle where the direction of a tilt of 0degrees, namely, an upward direction with respect to the Y axisdirection, when the rear side camera 212 is rotated around the X axis isdefined as a positive direction. Since image pickup is carried outusually setting the rear side camera 212 to a horizontal position, theelevation angle φ is 0 degrees. However, in order to pick up a panoramicimage of the celestial sphere, it is necessary to successively pick upan image of a subject by successively tilting the camera to change theelevation angle φ.

FIGS. 6(a) to 6(c) are views illustrating a panoramic image picked upwhen the initial position of the rear side camera 212 of the portableterminal 200 is in the direction of the orientation angle θ.

As depicted in a top plan view of FIG. 6(a), the rear side camera 212 atthe initial position is directed to the direction 220 of the orientationangle θ, and the elevation angle φ of the rear side camera 212 is 0degrees as depicted in a side elevational view of FIG. 6(b). Anomnidirectional panoramic image at the elevation angle φ=0 degrees ispicked up while the pan angle of the rear side camera 212 issuccessively changed within the range from −180 degrees to +180 degreeswith respect to the reference direction 220 with the elevation angle φkept to 0 degrees. FIG. 6(c) depicts a panoramic image 300 picked up inthis manner. The center of the panoramic image 300 is at the pan angleof 0 degrees, and the left half of the panoramic image 300 is an imagepicked up while the pan angle is successively changed from 0 degrees to−180 degrees, and the right half of the panoramic image 300 is an imagepicked up while the pan angle is successively changed from 0 degrees to+180 degrees.

Since the position of the center of the pan angle of 0 degrees of thepanoramic image 300 is displaced by the orientation angle θ to the eastfrom the true north, the position of the north (N), south (S), east (E)and west (W) are such as those indicated by broken lines. Only if thepanoramic image 300 has, as information relating to the image pickuporientation, the orientation angle θ of the position of the center ofthe pan angle of 0 degrees, pixel positions of the north (N), south (S),east (E) and west (W) can be determined by calculation taking thedisplacement of the orientation angle θ into consideration.Alternatively, in place of the orientation angle θ, coordinate values ofthe pixel positions of the north (N), south (S), east (E) and west (W)may be used as information relating to the image pickup direction.

In order to obtain a panoramic image of the celestial sphere, it isnecessary to carry out image pickup by successively changing theelevation angle of the rear side camera 212. For example, if it isassumed that the angle of view of the rear side camera 212 is 60degrees, then a panoramic image of the celestial sphere can be obtainedin principle if the rear side camera 212 is tilted by ±60 degreesupwardly and downwardly and, in this state, similar image pickup iscarried out successively changing the pan angle within the range from−180 degrees to +180 degrees.

FIGS. 7(a) to 7(c) are views illustrating a panoramic image picked upwhen the elevation angle φ of the rear side camera 212 is 60 degrees. Asdepicted in a top plan view of FIG. 7(a), the rear side camera 212 atthe initial position is directed to the direction 220 of the orientationangle θ, and the elevation angle φ of the rear side camera 212 is 60degrees as depicted in a side elevational view of FIG. 7(b). Such apanoramic image 302 at the elevation angle φ=60 degrees as depicted inFIG. 7(c) is picked up while the pan angle of the rear side camera 212is successively changed within the range from −180 degrees to +180degrees with respect to the reference direction 220 with the elevationangle φ of the rear side camera 212 kept to 60 degrees.

Similarly, a panoramic image at the elevation angle φ=−60 degrees ispicked up while the pan angle is successively changed within the rangefrom −180 degrees to +180 degrees with the elevation angle φ of the rearside camera 212 kept to −60 degrees. If the panoramic images at theelevation angle φ=0 degrees, 60 degrees and −60 degrees are combined,then a panoramic image of the celestial sphere can be obtained. However,in actual incorporation, in order to correct non-alignment originatingfrom distortion of the lens when images are stuck together in a boundaryregion of the angle of view, a method of picking up images such thatportions thereof in the proximity of the boundary are overlapped witheach other is adopted frequently.

The celestial sphere panoramic image obtained in this manner hasinformation of an orientation angle and an elevation angle addedthereto, and the orientation and the elevation angle can be specifiedwith regard to an arbitrary pixel of the panoramic image based on theinformation. Further, latitude and longitude information measured by theGPS may be added as position information of the image pickup place tothe panoramic image. The additional information to be added to thepanorama information may be recorded in conformity with a standard foran image file called Exif (Exchangeable Image File Format) as anexample. It is possible to record the place name of the image pickupplace into part of a file name and record the image pickup date andtime, latitude and longitude, height, orientation angle of the imagepickup place and so forth may be recorded as data of the Exif format. Itis to be noted that, since the elevation angle is not defined in theExif format at present, it is recorded as extended data.

FIGS. 8A and 8B are views illustrating methods of generating a panoramicimage by connecting a plurality of images.

In the example of FIG. 8A, seven images 341 to 347 picked up while therear side camera 212 is successively tilted (or panned) are first mappedcylindrically and then are connected to each other to generate acylindrical image 340. When the images are connected to each other,portions of the images in the proximity of the boundaries of the imagesare overlapped with each other.

If the rear side camera 212 is successively panned (or tilted) to pickup images, then a plurality of cylindrical images depicted in FIG. 8Aare obtained in the panning direction (or tilting direction) as depictedin FIG. 8B. If the cylindrical images 340 a to 340 f are synthesizedsuch that portions thereof in the proximity of the boundaries thereofare overlapped with each other, then an omnidirectional panoramic image360 is obtained finally.

FIG. 9 is a functional block diagram of the panoramic image generationapparatus 100. Some or all of the functional components of the panoramicimage generation apparatus 100 can be incorporated by hardware, softwareor a combination of hardware and software of the portable terminal 200.Some of the functional components may be incorporated in a server whilesome of the functional components are incorporated in a client such thatthey are incorporated as a server-client system with a networkinterposed therebetween.

A three-axis gyro sensor 62 detects an angular velocity of the portableterminal 200 in the three directions of the X axis, Y axis and Z axisand supplies the detected angular velocities to a posture detection unit60. A three-axis acceleration sensor 64 detects an acceleration of theportable terminal 200 in the three directions of the X axis, Y axis andZ axis and supplies the detected accelerations to the posture detectionunit 60. The posture detection unit 60 uses a result of the detection bythe three-axis gyro sensor 62 and/or the three-axis acceleration sensor64 to detect three-axis components of an inclination of the portableterminal 200, namely, a tilt angle, a roll angle and a pan angle. Theposture detection unit 60 supplies the detected tilt angle, roll angleand pan angle of the portable terminal 200 as posture information to apan angle-elevation angle calculation unit 70.

Although the portable terminal 200 may incorporate only one of thethree-axis gyro sensor 62 and the three-axis acceleration sensor 64, ifboth of them are incorporated, then the detection accuracy of theposture information can be improved because they make up forshortcomings. Although the three-axis gyro sensor 62 can measure alsoduring movement, it has a shortcoming that a drift error is successivelyaccumulated. Although the three-axis acceleration sensor 64 can detect aposture angle from the gravitational acceleration when it is in astationary state, an error appears therewith during movement thereof.

Therefore, the posture detection unit 60 carries out, for example,weighted averaging of outputs of the three-axis gyro sensor 62 and thethree-axis acceleration sensor 64 to improve the detection accuracy.When a weighted average is calculated, during movement, the weight tothe detection component of the three-axis gyro sensor 62 is increasedwith respect to the detection component of the three-axis accelerationsensor 64. On the other hand, when it is regarded that the three-axisgyro sensor 62 and the three-axis acceleration sensor 64 aresubstantially in a stationary state, the weight to the detectioncomponent of the three-axis acceleration sensor 64 is increased withrespect to the weight to the detection component of the three-axis gyrosensor 62. Consequently, in a stationary state, a yaw drift by thethree-axis gyro sensor 62 is corrected by an output of the three-axisacceleration sensor 64, but during movement, the output of thethree-axis gyro sensor 62 is relied upon rather than the output of thethree-axis acceleration sensor 64, whereby posture information of ahigher degree of accuracy can be detected.

As another method, also it is possible for the posture detection unit 60to use a result of detection by a three-axis geomagnetic sensor 68 todetect three-axis components of an inclination of the portable terminal200. If the geomagnetic vector is tracked by the three-axis geomagneticsensor 68 to detect rotation of the portable terminal 200 around the Xaxis, Y axis and Z axis, then three-axis components of an inclination ofthe portable terminal 200, namely, a tilt angle, a roll angle and a panangle, can be detected. If the three-axis geomagnetic sensor 68incorporated in the portable terminal 200 is inclined, then the detecteddirection of the geomagnetic vector changes. If the observation spotdoes not vary by a great amount, then since the geomagnetic vector isdirected to a fixed direction, a variation of the posture of theportable terminal 200 from its initial position can be detected. If thethree-axis acceleration sensor 64 is used additionally, then since thevertical direction can be detected, also the initial posture can bedetected.

if the angle of rotation of the three-dimensional geomagnetic vectordetected by the three-axis geomagnetic sensor 68 per unit time ismeasured, then an angular velocity can be determined. Therefore, thethree-axis geomagnetic sensor 68 can be used in place of the gyrosensor. This is called magnetic gyro function. Further, if the variationangle from the initial position is determined, then since theinclination can be detected, the three-axis geomagnetic sensor 68 can beused as a three-axis posture sensor. However, when the geomagneticvector coincides with the pan axis or the tilt axis, an angle ofrotation around the axis cannot be determined.

If the three-axis gyro sensor 62 or the three-axis acceleration sensor64 is used in combination with the three-axis geomagnetic sensor 68,then both of an orientation and a posture can be measured with a highdegree of accuracy.

As described above, the posture detection unit 60 can determinethree-axis components of an inclination of the portable terminal 200 byany of the three-axis gyro sensor 62 by itself, the three-axisacceleration sensor 64 by itself, the three-axis geomagnetic sensor 68by itself, a combination of the three-axis gyro sensor 62 and thethree-axis acceleration sensor 64, a combination of the three-axisgeomagnetic sensor 68 and the three-axis gyro sensor 62, a combinationof the three-axis geomagnetic sensor 68 and the three-axis accelerationsensor 64, and a combination of the three-axis gyro sensor 62,three-axis acceleration sensor 64 and three-axis geomagnetic sensor 68.

The three-axis geomagnetic sensor 68 detects components of thegeomagnetic vector in the X-axis, Y-axis and Z-axis directions andsupplies the detected components to an orientation detection unit 66.The orientation detection unit 66 determines an orientation vector,which is a horizontal component of the geomagnetic vector, based on thegravity vector detected by the three-axis gyro sensor 62 or thethree-axis acceleration sensor 64 and the geomagnetic vector detected bythe three-axis geomagnetic sensor 68. Then, the orientation detectionunit 66 supplies the components of the orientation vector as orientationinformation to the pan angle-elevation angle calculation unit 70.

The pan angle-elevation angle calculation unit 70 determines, based onthe posture information of the portable terminal 200 supplied theretofrom the posture detection unit 60, a pan angle and an elevation anglewhich indicate the image pickup direction of the camera of the portableterminal 200, and supplies the determined pan angle and elevation angleto a component image generation unit 20. If orientation information issupplied from the orientation detection unit 66, then the panangle-elevation angle calculation unit 70 can associate the pan angle ofthe camera with the absolute orientation.

A GPS receiver 74 receives latitude and longitude, height and timeinformation from the GPS and supplies the received information to aposition detection unit 72. The position detection unit 72 supplies thelatitude and longitude information to the component image generationunit 20. The incorporation of the GPS receiver 74 is optional.

An image pickup unit 10 controls the front side camera 210 and/or therear side camera 212 to pick up an image of a subject and records thepicked up images into a frame memory 12.

An image pickup timing decision unit 14 decides an appropriate imagepickup timing by the image pickup unit 10 and instructs the image pickupunit 10 of a timing at which the shutter is to be released. The imagepickup timing decision unit 14 decides based on a sensor output of thethree-axis gyro sensor 62 and/or the three-axis acceleration sensor 64whether or not there is a movement of the camera. Further, the imagepickup timing decision unit 14 detects an inter-frame difference of amoving picture picked up by the image pickup unit 10 and decides whetheror not a subject which is moving exists.

The image pickup timing decision unit 14 instructs, in the case of anautomatic image pickup mode, the image pickup unit 10 to pick up animage at a timing at which it is decided that the camera exhibits nomovement and a subject which is moving does not exist. In the case of asemiautomatic image pickup mode, the image pickup timing decision unit14 omits the detection of a moving body and instructs the image pickupunit 10 to pick up an image at a timing at which it is decided that thecamera exhibits no movement. In the case of a manual image pickup mode,the image pickup timing decision unit 14 allows the user to release theshutter without instructing the image pickup unit 10 of an image pickuptiming.

The component image generation unit 20 stores a pan angle (or anabsolute orientation angle) and an elevation angle of the camerasupplied thereto from the pan angle-elevation angle calculation unit 70as a component image into a component image storage unit 22 in anassociated relationship with a picked up image stored in the framememory 12. If position information is supplied from the positiondetection unit 72, then the component image generation unit 20 furtherassociates the position information with the picked up image.

A panoramic image generation unit 30 stitches a plurality of componentimages stored in the component image storage unit 22 to synthesize apanoramic image and stores the panoramic image into a panoramic imagestorage unit 32.

A display unit 40 reads out, upon panoramic image pickup, a plurality ofpicked up component images before synthesis into a panoramic image fromthe component image storage unit 22 and displays the read out componentimages on a display 240. Consequently, the user can confirm on thedisplay 240 that there remains a region in which no image is picked upas yet in order to complete a panoramic image.

On the other hand, the display unit 40 reads out a synthesized panoramicimage from the panoramic image storage unit 32 and displays thepanoramic image on the display 240. A communication unit 80 transmitsthe completed panoramic image to a communication partner or uploads thecompleted panoramic image to a server.

A non-image pickup region decision unit 24 decides, based on informationof the pan angle and the elevation angle of picked up component imagesstored in the component image storage unit 22, a region for a componentimage which is not enough to complete a panoramic image. The non-imagepickup region decision unit 24 supplies information which specifies aregion for a component image not picked up as yet which is adjacent acomponent image picked up most recently to an image pickup guide portion52 of a user interface unit 50.

The image pickup guide portion 52 of the user interface unit 50generates a guide for guiding the user to subsequently pick up an imagein a region for a non-picked up component image which is designated bythe non-image pickup region decision unit 24 and causes the display unit40 to display the guide.

FIG. 10 is a flow chart illustrating a procedure of image pickup of apanoramic image.

The image pickup timing decision unit 14 executes an image pickup timingdecision process (S10). The image pickup unit 10 carries out imagepickup at an image pickup timing designated by the image pickup timingdecision unit 14 or an image pickup timing designated by the user andrecords the picked up image into the frame memory 12 (S12).

The posture detection unit 60 acquires posture information of theportable terminal 200, namely, a tilt angle, a roll angle and a panangle (S14).

The component image generation unit 20 stores the posture information ofthe portable terminal 200 as a component image in an associatedrelationship with the picked up image into the component image storageunit 22 (S16). The posture information of the portable terminal 200 maybe converted into a pan angle and an elevation angle of the rear sidecamera 212 and associated with the picked up image. In both cases, if aplurality of component images are stitched, then a panoramic image canbe synthesized. The component images may be connected based on theposture information of the portable terminal 200 or the information ofthe pan angle and the elevation angle of the rear side camera 212.

The non-image pickup region decision unit 24 decides from the postureinformation or the information of the pan angle and the elevation angleof a plurality of picked up component images stored in the componentimage storage unit 22 whether or not there remains a region in which noimage is picked up as yet and which is not enough to complete a finalpanoramic image (S18). If there remains no region in which no image ispicked up as yet, then the image pickup is ended (N at S18).

If there remains a region in which no image is picked up as yet (Y atS18), then the image pickup guide portion 52 causes an arrow mark to bedisplayed on the screen or issue an audio guide in order to guide theuser so that a component image which is not picked up as yet andneighbors with a component image picked up most recently is picked upnext (S20).

FIG. 11 is a flow chart illustrating a detailed procedure of the imagepickup timing decision process at step S10 by the image pickup timingdecision unit 14.

It is decided which one of the automatic, semiautomatic and manual modesthe image pickup mode is (S30). If the image pickup mode is the manualimage pickup mode, then the user would release the shutter to instructthe image pickup unit 10 of an image pickup timing (S40).

If the image pickup mode is the automatic image pickup mode, then amovement of the camera is detected based on outputs of the three-axisgyro sensor 62 and the three-axis acceleration sensor 64 (S32). Here,the “movement of the camera” includes vibration of the camera or camerashake when the image pickup angle is changed. If the time variation ofthe inclination data detected by the three-axis gyro sensor 62 issmaller than a threshold value TA and besides the time variation of theinclination data detected by the three-axis acceleration sensor 64 issmaller than a threshold value TB, then it is decided that there is nomovement of the camera (N at S32), and the processing advances to stepS34. In any other case, it is decided that there is a movement of thecamera (Y at S32), and it is waited that the movement of the cameradisappears.

Then, moving body detection is carried out (S34). If the absolutedifference between frames is smaller than a threshold value TC, it isdecided that no subject which is moving is caught in the image (N atS34), and the image pickup timing decision unit 14 instructs the imagepickup unit 10 of this point of time as an image pickup timing (S38). Ifthe absolute difference between the frames exceeds the threshold valueTC (Y at S34), then the movement detection of the camera at step S32 andthe moving body detection at step S34 are repeated.

The moving body detection process at step S34 is preferably configuredsuch that, although a moving body such as a person who is walking at aremote place is permitted, a moving body in a near view is notpermitted. This is because, if a moving body is in the near view, thenthis causes inconvenience in synthesis of a panoramic image. Therefore,when a region in which a movement is detected from an inter-framedifference is greater than a predetermined size, it may be decided thata moving body is caught, but in any other case, it may be decided thatno moving body is caught.

If a moving body is detected, then permission to carry out image pickupmay be sought to the user. For example, either a message such as, forexample, “Something is moving. If image pickup is to be continued withthis, then please depress the determination button.” is displayed oroutputted by voice, and if the user issues an image pickup instruction,then image pickup may be carried out even if a moving body is detected.If the user does not permit image pickup, then image pickup is carriedout after it is waited that a moving body is not detected any more.

When the image pickup mode is the semiautomatic image pickup mode, theimage pickup timing decision unit 14 carries out movement detection ofthe camera at step S32, but skips the moving body detection at step S34.The image pickup timing decision unit 14 instructs the image pickup unit10 of the point of time at which it is decided that there is no movementof the camera as an image pickup timing (S38).

FIGS. 12(a) to 12(c) are views illustrating panoramic image pickupguides by the image pickup guide portion 52.

The panoramic image 360 is formed from sectoral component imagesoverlapping with each other as illustrated in FIGS. 8A and 8B. However,here the whole celestial sphere is simply divided into grid-like cellsin accordance with the angle of view of the camera and component imagesare represented by the cells for simplified description.

FIG. 12(a) illustrates a method wherein pickup of a component image iscarried out while the pan angle is successively changed within a rangefrom approximately −180 degrees to approximately +180 degrees with theelevation angle of the camera kept fixed is repeated changing theelevation angle to pick up an image of the whole celestial sphere. Inthis image pickup method, since the yaw drift of the three-axis gyrosensor 62 is accumulated, errors accumulate in the image pickupdirection.

In contrast, FIG. 12(b) illustrates another method wherein pickup of acomponent image is carried out while the elevation angle is successivelychanged within a range from approximately −90 degrees to approximately+90 degrees with the pan angle kept fixed is repeated changing the panangle to pick up an image of the whole celestial sphere. After componentimages are picked up with the elevation angle changed from approximately−90 degrees to approximately +90 degrees at a certain pan angle, at anext pan angle, component images are picked up while the elevation angleis successively changed from approximately +90 degrees to approximately−90 degrees as indicated by arrow marks. By the method, the continuityof image pickup regions can be assured at a turn of the pan angle. Ifimage pickup is carried out in the order of the arrow marks of FIG.12(b), then since the pan angle merely makes one rotation, the drifterror of the three-axis gyro sensor 62 is not accumulated, and anaccurate image pickup direction can be detected.

Where both of the front side camera 210 and the rear side camera 212 areused, while the front side camera 210 picks up a component image whilesuccessively changing the elevation angle from approximately −90 degreesto approximately +90 degrees, the rear side camera 212 can pick upcomponent images at pan angles different by 180 degrees successivelychanging the elevation angle from approximately +90 degrees to −90degrees as indicated by arrow marks in FIG. 12(c).

FIGS. 13(a) and 13(b) are views illustrating an example of componentimages and a panoramic image. Also here, description is given assumingthat component images are grid-like cells. FIG. 13(a) depicts an exampleof grid-like component images, and FIG. 13(b) depicts an example of apanoramic image synthesized by stitching the component images.

FIG. 14 is a view illustrating the panoramic image during image pickup.Component images picked up already are displayed, and blank grids areregions for component images which are not picked up as yet. As acomponent image to be picked up next, a region 271 for a component imageadjacent a component image 270 picked up most recently is selected fromwithin a region which includes no picked up image. Since the order ofcomponent images to be picked up is preferably such that componentimages are picked up with the elevation angle successively changed whilethe pan angle is kept fixed as illustrated in FIG. 12(b), the guide forthe image pickup order of component images is such as that indicated byan arrow mark.

FIG. 15 illustrates an example of the guide for an image pickup order ofa panoramic image pick up. In a right lower region of the screen of thedisplay 240, the panoramic image of FIG. 14 being picked up currently isdisplayed, and a region for a component image to be picked up next isindicated by an arrow mark. Consequently, the user can discriminate atwhich point the image pickup of the panoramic image arrives at present.

In a left region of the screen of the display 240, an image 272 beingpicked up at present by the camera is displayed and plays a role of afinder. A region to be picked up next is indicated by a large arrow mark273. An image pickup timing is designated automatically or manually.After the image pickup ends, the user would pan or tilt the camera inthe direction of the arrow mark to enter image pickup for a nextcomponent image.

Based on detection results of the moving direction and the moving speedby the three-axis gyro sensor 62 and/or the three-axis accelerationsensor 64, a message is displayed on the screen like “Please move thecamera slowly in the direction of the arrow mark.” The message may beoutputted by voice. If the moving speed of the camera is excessivelyhigh, then the message or the arrow mark may be changed in color for thewarning. On the other hand, if the position of the camera moves, thenthe center of rotation of the camera is displaced and causes an error.When a variation of the position of the camera is detected byself-position estimation as described in the description of anembodiment 2 hereinafter described, such a warning message as “Thecenter of rotation of the camera has displaced!” is displayed.

FIG. 16 is a schematic view of a see-through type head-mounted display(HMD) 250. This is an application example wherein the functionalconfiguration depicted in FIG. 9 is mounted in the see-through type HMD250. The see-through type HMD 250 includes a camera, a three-axis gyrosensor and/or a three-axis acceleration sensor, a three-axis geomagneticsensor, and a display 252. The user would wear the see-through type HMD250 like an eyeglass and carry out panoramic image pickup while viewingan outside world which can be seen through the display 252 and an imagedisplayed on the display 252.

FIGS. 17(a) and 17(b) are views illustrating examples of a screen imageof the display 252 of the see-through type HMD 250. As shown in FIG.17(a), an image pickup region 254 indicated by solid lines is locatedcentrally of the display 252, and an image of the image pickup region254 is picked up and stored as a component image with which a pan angleand an elevation angle of the camera are associated. A guide region 256indicated by broken lines is a region in which an image is to be pickedup next. A leftwardly directed arrow mark 258 is displayed on the screenso that the user may turn the head to the left and face in the directionof the guide region 256.

FIG. 17(b) depicts an example of a screen image of the display 252 whenthe user turns the head to the left. The image pickup region 254indicated by solid lines and the guide region 256 indicated by brokenlines are located centrally of the screen. The user would move the headso that the quadrangle (image pickup region 254) of solid linesindicated on the display 252 and the quadrangle (guide region 256) ofbroken lines may coincide with each other. When the guide region 256 andthe image pickup region 254 almost overlap with each other, image pickupis carried out automatically and a component image is acquired. Aplurality of component images picked up already may be displayed in ajuxtaposed relationship in a corner of the display 252 so that the usercan grasp to which degree the image pickup of a panoramic image hasprogressed.

As described above, with the panoramic image generation apparatus of thepresent embodiment, by stitching picked up images using a three-axisgyro sensor and/or a three-axis acceleration sensor, a panoramic imagecan be synthesized simply. If a three-axis geomagnetic sensor is usedadditionally, then an absolute orientation can be associated with thepanoramic image. Consequently, even if an expensive pan-tilt camera witha camera platform is not used, a panoramic image can be picked simply ata low cost by the portable terminal 200 with a camera.

Since an existing synthesis method of a panoramic image uses imagematching, when characteristic points of images cannot be associated witheach other upon stitching, it is difficult to synthesize a panoramicimage. Especially, in the case of a subject which does not include acharacteristic point in an image like the sky or a white wall and doesnot allow association, stitching by image matching is impossible. Inthis regard, with the panoramic image generation apparatus of thepresent embodiment, even if an image does not have a characteristicpoint, since a pan angle and an elevation angle of the camera from thethree-axis gyro sensor and/or the three-axis acceleration sensor can beassociated with a picked up image, a panoramic image of the celestialsphere can be synthesized accurately.

However, since an error is involved in posture detection by thethree-axis gyro sensor and/or the three-axis acceleration sensor, when apanoramic image is to be synthesized finally, if image matching isapplied additionally, then a panoramic image of high picture quality canbe generated.

Further, according to existing panorama synthesis by image matching,since time is required for image processing, it is difficult to decide anon-image pickup region of a panoramic image on the real time basiswhile image pickup is being carried out. In this regard, with thepanoramic image generation apparatus of the present embodiment, even ifimage matching is not applied, since stitching of picked up images canbe carried out based on posture information obtained from the three-axisgyro sensor and/or the three-axis acceleration sensor, it is possible todetect a non-image pickup region on the real time basis and guide a nextimage pickup region. Also it is possible to guide the user so that theuser may move the portable terminal 200 slowly based on a moving speeddetected by the three-axis gyro sensor and/or the three-dimensionalacceleration sensor.

Conventionally, if it is tried to pick up a panoramic image while acamera of a portable terminal or the like is being panned and/or tilted,then it is difficult to generate an accurate panoramic image from amovement of the camera or displacement of the axis of rotation. However,with the panoramic image generation apparatus of the present embodiment,since a movement of the camera is detected by the three-axis gyro sensorand/or the three-axis acceleration sensor and the shutter isautomatically released to pick up an image when there is no movement ofthe camera, even in panoramic image pickup by the portable terminal 200,a bad influence of a movement of the camera can be suppressed. Further,if also moving body detection is carried out additionally and imagepickup is carried out at a timing at which no moving body is detected,then it is possible to acquire picked up images which include no movingperson or body to generate a panoramic image.

Embodiment 2

FIG. 18 is a functional block diagram of a panoramic image generationapparatus 100 according to an embodiment 2. The panoramic imagegeneration apparatus 100 of the embodiment 2 includes, in addition tothe components of the panoramic image generation apparatus 100 of theembodiment 1 described hereinabove, a characteristic point extractionunit 82, a self-position estimation unit 84 and a camera positiondisplacement decision unit 86, and has a function of estimating thecamera position of the portable terminal 200 to cause the user to adjustthe displacement of the camera position. The other components andfunctions are same as those of the panoramic image generation apparatus100 of the embodiment 1, and therefore, description of them is omittedherein.

The characteristic point extraction unit 82 extracts characteristicpoints between adjacent ones of a plurality of component images storedin the component image storage unit 22 and different in a pan angle or atilt angle from each other to detect control points for associating theadjacent component images with each other. The characteristic pointextraction unit 82 supplies information of the characteristic pointsassociated with each other between adjacent component images to theself-position estimation unit 84. Further, the characteristic pointextraction unit 82 supplies the detected control points to the panoramicimage generation unit 30. The control points are all or some of thecharacteristic points associated with each other between adjacentcomponent images.

FIGS. 21(a) and 21(b) are views depicting control points detectedbetween two adjacent component images 401 and 402. Here, nine controlpoints are detected.

FIG. 22 is a view illustrating a manner in which the two adjacentcomponent images 401 and 402 are aligned and combined with each otherbased on the control points. The panoramic image generation unit 30rotates the component image 402 with respect to the component image 401to combine them so that the control points (indicated by round marks inFIG. 22) of the two images coincide with each other as far as possiblein an overlapping region of the adjacent component images 401 and 402.

The self-position estimation unit 84 simultaneously estimates, based onthe information of two-dimensional coordinates, on the two adjacentcomponent images, of the characteristic points of the images,three-dimensional coordinates of the characteristic points andcoordinates of the camera positions at which the individual componentimages are picked up. Since the camera positions when two componentimages are picked up are displaced from each other, the two componentimages are parallax images from different visual points. Therefore, thetwo component images include depth information in the form of aparallax, and by solving simultaneous equations which satisfy eachcharacteristic point, three-dimensional coordinates of thecharacteristic point and three-dimensional coordinates (camera position)of the visual point of the component image can be determinedsimultaneously. As one of methods of simultaneously determiningthree-dimensional position information of a subject and a visual pointposition based on a plurality of parallax images whose visual pointpositions are different from each other, a technology called SLAM(Simultaneous Localization and Mapping) is available.

The SLAM is a technology of carrying out self-position estimation andenvironment mapping simultaneously based on information acquired fromsensors and is applied to an autonomous mobile robot and so forth. Theself-position estimation unit 84 estimates a three-dimensional positionof a characteristic point of a subject and a three-dimensional positionof a camera using the publicly known SLAM technology as an example.However, some other self-position estimation technology may be used. TheSLAM is introduced, for example, in the following paper. Andrew J.Davison, “Real-Time Simultaneous Localisation and Mapping with a SingleCamera,” ICCV 2003.

The self-position estimation unit 84 supplies the information of thethree-dimensional coordinate values of the characteristic points of thesubject and the three-dimensional coordinate values of the camerapositions obtained by the estimation to the component image generationunit 20. The component image generation unit 20 stores the postureinformation of the portable terminal 200, the three-dimensional positioninformation of the characteristic points of the subject and thethree-dimensional position information of the camera positions in anassociated relationship with the component images into the componentimage storage unit 22.

Since a component image includes three-dimensional position informationof a subject, if the panoramic image generation unit 30 combines aplurality of component images to synthesize a panoramic image, then athree-dimensional panoramic image having the three-dimensional positioninformation of the subject is obtained. If the three-dimensionalpanoramic image is expanded on a three-dimensional coordinate system,then a simple three-dimensional model of the subject can be generated.Further, since depth information of the subject is obtained, also it ispossible to generate a stereo panoramic image.

The camera position displacement decision unit 86 decides whether or notthe camera position estimated by the self-position estimation unit 84 isdisplaced from the first position (origin). If the camera position isdisplaced, then an accurate synthesized panoramic image is not obtained.The Image pickup guide portion 52 causes a locus of movement of thecamera position to be displayed on the display based on thethree-dimensional coordinate values of the camera position estimated bythe self-position estimation unit 84 so that the user can visuallyrecognize the camera position at the timing of panoramic image pickup.Further, if it is decided by the camera position displacement decisionunit 86 that the camera position is displaced, then the Image pickupguide portion 52 guides the user to adjust the camera position orinstructs the user to retry image pickup.

However, even if the camera position is displaced a little, if thecomponent image is corrected so that the camera position is returned tothe origin, then a panoramic image can be synthesized. Therefore, whenthe panoramic image generation unit 30 combines component images, ituses the three-dimensional position information of the characteristicpoints associated with the component images and the three-dimensionalposition information of the camera positions to adjust thethree-dimensional coordinate values of the characteristic points of thecomponent images so that the displacement of the camera position fromthe origin may approach zero thereby to correct the component images.Then, the panoramic image generation unit 30 combines the correctedcomponent images to generate a panoramic image.

FIG. 19 is a flow chart illustrating a procedure of image pickup of apanoramic image by the panoramic image generation apparatus 100according to the embodiment 2. Processes at steps S10, S12, S14, S18 andS20 of FIG. 19 are same as the processes at steps S10, S12, S14, S18 andS20 by the panoramic image generation apparatus 100 according to theembodiment 1 depicted in FIG. 10, and therefore, a description of themis omitted herein.

After posture information of the portable terminal 200, namely, a tiltangle, a roll angle and a pan angle, is acquired at step S14, theinformation is used to carry out a self-position estimation process atstep S15. Position information of the camera of the portable terminal200 is obtained by the self-position estimation process, and adjustmentof the camera position is carried out as occasion demands. Further,three-dimensional position information of the characteristic points ofthe subject is obtained by the self-position estimation process.

At step S17, the component image generation unit 20 stores the postureinformation of the portable terminal 200, the three-dimensional positioninformation of the characteristic points of the subject and thethree-dimensional position information of the camera position in anassociated relationship with the picked up image as a component imageinto the component image storage unit 22.

FIG. 20 is a flow chart illustrating a detailed procedure of theself-position estimation process at step S15 of FIG. 19.

The characteristic point extraction unit 82 extracts, in an overlappingregion of two adjacent component images, characteristic points of theobject whose image is picked up and carries out matching of them (S50).The self-position estimation unit 84 uses the SLAM to estimatethree-dimensional coordinate values of the position of the camera atwhich each component image is picked up and three-dimensional coordinatevalues of the characteristic points of the object (S52).

The camera position displacement decision unit 86 decides whether or notthe camera position upon image pickup is displaced exceeding apredetermined threshold value from the first position (S54). If thedisplacement of the camera position remains within the threshold value(N at S54), then the self-position estimation process is ended. Here,the user may be permitted to input an instruction indicating that whichone of an image of a distant view and an image of a near view is to bepicked up, and the threshold value for the decision of the displacementof the camera position may be changed over depending upon whether animage of a distant view or an image of a near view is to be picked up.In the case of image pickup of a distant view, the permissible range forthe displacement of the camera position is greater in comparison withthat in image pickup of a near view. Therefore, for image pickup of adistant view, the threshold value is set higher than the threshold valuefor image pickup of a near view.

If a displacement of the camera position exceeding the predeterminedthreshold value is detected (Y at S54), then the camera positiondisplacement decision unit 86 instructs the image pickup guide portion52 to adjust the camera position. The image pickup guide portion 52either instructs the user to correct the camera position or invalidatessome or all of component images picked up already and instructs the userto retry image pickup (S56). Thereafter, the processing returns to S10.

FIG. 23 is a view illustrating an example of a guide for urging the userto adjust the displacement of the camera position. The image pickupguide portion 52 three-dimensionally displays the camera position uponpanoramic image pickup at a portion of the screen of the display. Theimage pickup guide portion 52 sets a first camera position 410 a to theorigin and displays histories 410 b, 410 c and 410 d of the cameraposition in a three-dimensional coordinate system.

The user can view the camera position at present or the locus ofmovement of the camera position displayed on the display to confirmwhether or not the camera position is displaced from the first position.In addition to graphical display of the three-dimensional position ofthe camera and the posture information of the portable terminal 200 onthe display unit, the direction of movement of the camera position maybe presented by voice or a text. Alternatively, the portable terminal200 may be vibrated to notify the user that the camera position isdisplaced or is displaced exceeding a permissible range.

In this three-dimensional coordinate system, an inner side sphere 400centered at the origin corresponds to a first threshold value Th1 forthe displacement of the camera position, and an outer side sphere 402centered at the origin corresponds to a second threshold value Th2(>Th1) for the displacement of the camera position. The first thresholdvalue Th1 is an upper limit to a permissible displacement amount withinwhich component images can be stitched to generate a panoramic imagewithout correction. The second threshold value Th2 is an upper limit toa permissible displacement amount within which, if component images arecorrected, the component images after correction can be stitched togenerate a panoramic image.

If the camera position at the time of starting of image pickup isrepresented by X0 and the camera position upon image pickup of an nthcomponent image is represented by Xn, then when |Xn−X0|<Th1, a componentimage is picked up at the camera position. Here, |Xn−X0| represents thedistance from the camera position X0 upon starting of the image pickupto the camera position Xn upon image pickup of the nth component image.If this is described with reference to FIG. 23, then when the cameraposition at present is within the inner side sphere 400, a componentimage is picked up at the camera position, and a correction instructionfor the camera position is not issued.

When |Xn−X0|≧Th1 and |Xn−X0|<Th2, since a panoramic image can besynthesized by correction, a component image is picked up at the cameraposition. However, a notification that the camera position is displacedis issued to the user to urge the user to correct the camera position.In regard to FIG. 23, when the camera position at preset is between theinner side sphere 400 and the outer side sphere 402, a component imageis picked up at the camera position, but a correction instruction forthe camera position is issued. At this time, the panoramic imagegeneration unit 30 corrects the component image based on thedisplacement of the camera position and then combines the correctedcomponent image to generate a panoramic image.

When |Xn−X0|≧Th2, since synthesis of a panoramic image is impossibleanymore even if correction is carried out, image pickup is not carriedout at the camera position, and the user is urged to move the cameraposition. Then, if the camera position enters back into the range inwhich image pickup is possible, then image pickup is started. Even if apredetermined period of time elapses, if the camera position does notenter back into the region in which image pickup is possible, then thepanoramic image pickup till then is canceled, and the user is instructedto retry panoramic image pickup.

The threshold values Th1 and Th2 for defining the range within whichimage pickup is possible are preferably made variable in response to thedistance from the point of view to the subject. The distance from thepoint of view to the subject can be determined from three-dimensionalcoordinate values of the characteristic points of the subject and thethree-dimensional coordinate values of the camera position estimated bythe self-position estimation unit 84. If the subject is a distant view,then even if the displacement of the camera position is great, thepositional displacement of the subject on the image is not great, andtherefore, the error when a panoramic image is synthesized is small.However, if the subject is a near view, then since a great amount ofpositional displacement of the subject on the image arises from a smallamount of displacement of the camera position, the error when apanoramic image is synthesized is great. Therefore, the threshold valuesTh1 and Th2 when the subject is a distant view can be set greater thanthe threshold values Th1 and Th2 when the subject is a near view. Thecamera position displacement decision unit 86 may adjust the thresholdvalues Th1 and Th2 in response to the distance from the camera positionto the subject.

While, in the foregoing description, the camera position is estimated bythe SLAM, the camera position may be acquired using position informationdetected by the GPS receiver 74.

As described above, with the panoramic image generation apparatus of thepresent embodiment, it is possible to estimate a camera position and athree-dimensional position of a subject whose image is picked up using atechnology such as the SLAM and notify a user of displacement of thecamera position. Where it is decided in response to the distance to thesubject whether or not the displacement of the camera position is withinthe permissible range and the displacement of the camera positionexceeds the permissible range, it is possible to urge the user to movethe camera position.

Even if the three-axis gyro sensor 62 or the three-axis accelerationsensor 64 can detect a posture of the portable terminal 200, it isdifficult to detect displacement of the camera position. In this regard,with the present embodiment, a camera position and a three-dimensionalposition of a subject can be estimated simultaneously using the factthat, if the camera position is displaced, then a parallax appearsbetween picked up component images. Therefore, it is possible to detectdisplacement of the camera position and urge the user to adjust thecamera position or correct the image based on the displacement of thecamera position.

Embodiment 3

FIG. 24 is a functional block diagram of a panoramic image generationapparatus 100 according to an embodiment 3. The panoramic imagegeneration apparatus 100 of the present embodiment has a function ofsetting a region of interest during panoramic image pickup. Componentscommon to those of the panoramic image generation apparatus 100 of theembodiment 1 are denoted by like reference numerals and the descriptionof them is omitted herein.

A POI/ROI setting unit 42 includes a region-of-interest setting portion44, a degree-of-interest setting portion 46 and a tag setting portion48. The POI/ROI setting unit 42 sets a point POI (point of interest) ofinterest or a region ROI (region of interest) of interest to an imagepicked up by the image pickup unit 10, gives a degree of interest and atag to a region of interest, and supplies region-of-interest informationincluding coordinate information, a degree of interest and taginformation of the region of interest to the component image generationunit 20. The component image generation unit 20 stores theregion-of-interest information in an associated relationship with thepicked up image into the component image storage unit 22.

The region-of-interest setting portion 44 sets a region designated by acamera operator detection portion 90 of the image pickup unit 10 or aregion designated by a touch panel inputting portion 56 of the userinterface unit 50 as a region of interest. The degree-of-interestsetting portion 46 sets a degree of interest designated by a facialexpression decision portion 92 of the image pickup unit 10 as a degreeof interest in the region of interest set by the region-of-interestsetting portion 44. The tag setting portion 48 sets a text designated bya voice recognition portion 54 of the user interface unit 50 as a tag tothe region of interest set by the region-of-interest setting portion 44.

A POI/ROI processing unit 58 reads out a component image from thecomponent image storage unit 22 and reads out a synthesized panoramicimage from the panoramic image storage unit 32. Then, the POI/ROIprocessing unit 58 carries out an image process such as emphasis,blurring, zooming or changing of the picture quality for the region ofinterest set to the component image or the panoramic image. Where adegree of interest is set to the region of interest, image processing inaccordance with the degree of interest is carried out. For example, thedegree of emphasis is changed or the picture quality is changed inresponse to the degree of interest.

Further, the POI/ROI processing unit 58 may protect a region of interestwhen a panoramic image is processed. For example, when an artificialobject such as an airship or a person is to be combined into a panoramicimage, the POI/ROI processing unit 58 protects the region of interest bydisposing the artificial object behind the region of interest or by notdisposing the artificial object in the region of interest so that theregion of interest may not be hidden by such artificial object.

The display unit 40 displays a component image or a panoramic image inwhich image processing is carried out for a region of interest by thePOI/ROI processing unit 58. Further, when the display unit 40 displays apanoramic image, it may scroll and display the panoramic image so thatthe region of interest may come to the center of the screen of thedisplay 240.

Further, when the display unit 40 changes over the display image fromone panoramic image to another panoramic image, it may scroll anddisplay the panoramic image after the changeover so that the region ofinterest after the changeover may come to the center of the screen ofthe display 240.

Specifically, it is assumed that a marker representative of a particularobject such as a building is set to a particular orientation of a firstpanoramic image and a panorama viewer is configured such that, if themarker is selected, then the display image is changed over to a secondpanoramic image which includes the particular object. Also on the secondpanoramic image, the particular object is frequently set to a region ofinterest. Accordingly, upon changeover of a panoramic image, if thechangeover is controlled such that a region of interest always comes tothe center of the screen, then also on the panoramic image of thedestination of the changeover, the object designated by the marker canbe viewed from the beginning. For example, if the Osaka Castle isincluded in the first panoramic image and a marker is set to the OsakaCastle, then even if the marker is selected to change over the displayto the second panoramic image, the Osaka Castle can be viewed on thesecond panoramic image. Since a panoramic image is a picked up image ofa landscape within a wide range, if a panoramic image is changed over,then the direction is sometimes lost sight of. However, if the region ofinterest is displayed preferentially on a panoramic image afterchangeover, then such a problem as described just above can be avoided.

Where a plurality of regions of interest/points of interest are set to apanoramic image, when the panoramic image is scrolled and displayed, theregions of interest may be displayed, for example, in a priority orderof a free region and a rectangular region. Alternatively, the regions ofinterest may be displayed in a priority order of the magnitude of thearea or in a priority order of the degree of interest. Alternatively, aregion of interest may be displayed preferentially to a point ofinterest. Further, when a marker of a first panoramic image isdesignated to change over the display image to a second panoramic imageassociated with the marker, the regions of interest/points of interestmay be displayed in order preferentially from that region orinterest/point of interest which is nearer to the direction of themarker of the first panoramic image.

The image pickup unit 10 includes the camera operator detection portion90 and the facial expression decision portion 92.

The camera operator detection portion 90 decides whether or not thecamera operator is caught in an image picked up by the image pickup unit10, and supplies, if the camera operator is caught, information fordesignating the region in which the camera operator is caught to theregion-of-interest setting portion 44 of the POI/ROI setting unit 42.The camera operator detection portion 90 can detect the camera operatorby regarding a person caught in a great size in the picked up image asthe camera operator. The region-of-interest setting portion 44 sets theregion in which the camera operator is caught as a region of interest.The POI/ROI processing unit 58 carries out a process for blurring theregion in which the camera operator is caught or fills the region with asurrounding texture to erase the region in which the camera operator iscaught.

In panoramic image pickup, when the portable terminal 200 is panned topick up an image, the camera operator is frequently caught. Further,when, while an image of the front is picked up by the rear side camera212 of the portable terminal 200, the front side camera 210 is used tosimultaneously pick up an image also on the opposite side different by180 degrees, the camera operator is frequently caught by the front sidecamera 210. In such cases, by detecting the region in which the cameraoperator is caught and setting the region as a region of interest, anunintended image of the camera operator can be erased from the panoramicimage. Alternatively, when the camera operator is caught intentionally,the region in which the camera operator is caught may be set as a regionof interest such that, when the POI/ROI processing unit 58 carries outan image process for emphasizing the region of interest or when thedisplay unit 40 displays a panoramic image, the display position of thepanoramic image may be adjusted so that the region of interest may bedisplayed at the center of the display unit.

The facial expression decision portion 92 decides the facial expressionof the camera operator whose image is picked up by the front side camera210 during panoramic image pickup by the rear side camera 212. Apublicly known facial expression recognition technology may be used forthe decision of the facial expression. If it is decided that the cameraoperator has such an affirmative facial expression upon panoramic imagepickup that the camera operator smiles or gazes to the subject, thefacial expression decision portion 92 determines a degree of interest ofa value, for example, from 0 to 100 in response to the facial expressionand supplies the degree of interest to the degree-of-interest settingportion 46 of the POI/ROI setting unit 42. When a region of interest isalready set by the region-of-interest setting portion 44, thedegree-of-interest setting portion 46 sets the degree of interestsupplied thereto from the facial expression decision portion 92 to theregion of interest. However, when a region of interest is not set asyet, the degree-of-interest setting portion 46 treats an entire pickedup image or an object in the proximity of the center of a picked upimage as a region of interest and sets a degree of interest. The POI/ROIprocessing unit 58 executes a process in accordance with the degree ofinterest for the region of interest of the picked up image. For example,the picture quality is raised as the degree of interest increases.

The user interface unit 50 includes the voice recognition portion 54 andthe touch panel inputting portion 56 and supports the user to set aregion of interest or set a tag to a picked up image.

The voice recognition portion 54 acquires voice of the user duringpanoramic image pickup from a microphone built in the portable terminal200, converts the voice into a text by a voice recognition process andsupplies the text to the tag setting portion 48. The tag setting portion48 sets the text as a tag to the region of interest. A tag may be set tothe region of interest while the voice is kept as it is withoutconversion into a text. The degree of interest may be set in response tothe magnitude of the sound volume. If a region of interest is not set,then the entire picked up image or an object in the proximity of thecenter of the picked up image is treated as a region of interest, and atag is set to the region of interest. Consequently, the user can add taginformation such as a name to the subject in which the user isinterested during panoramic image pickup.

The touch panel inputting portion 56 receives information fordesignating a region of interest of a picked up image from the user on atouch panel provided on the display 240 and supplies the information fordesignating the region of interest to the region-of-interest settingportion 44.

FIG. 25 is a flow chart illustrating a procedure of image pickup of apanoramic image by the panoramic image generation apparatus 100according to the embodiment 3.

The image pickup timing decision unit 14 executes an image pickup timingdecision process (S60). The image pickup unit 10 picks up an image atthe image pickup timing designated by the image pickup timing decisionunit 14 or at an image timing designated by the user and supplies thepicked up image to the frame memory 12 (S62). The processes at steps S60and S62 are same as the processes at steps S10 and S12, respectively, bythe panoramic image generation apparatus 100 in the embodiment 1depicted in FIG. 10.

The touch panel inputting portion 56 detects whether or not there is aninput which designates a region of interest from the user to the touchpanel of the display 240 on which the picked up image is displayed(S64). If there is a touch panel input which designates a region ofinterest (Y at S64), then the region-of-interest setting portion 44 setsa region of interest to the picked up image (S66). If there is no touchpanel input which designates a region of interest (N at S64), then theprocessing advances to step S68.

The facial expression decision portion 92 decides the facial expressionof the camera operation caught in the picked up image of the front sidecamera 210 (S68). If the facial expression is an affirmative one like asmiling face (Y at S68), then the degree-of-interest setting portion 46sets a degree of interest to the region of interest in response to theaffirmative degree of the facial expression (S70). If a region ofinterest is not set as yet, then the facial expression decision portion92 sets, as a region of interest, for example, an object located at thecenter of the picked up image or an object which occupies a greater areathan any other object in the image pickup image, and then sets a degreeof interest to the region of interest. If the facial expression is notan affirmative one like a smiling face (N at S68), the processingadvances to step S72.

The voice recognition portion 54 decides whether or not there is a voiceinput from the user during image pickup (S72). If there is a voice input(Y at S72), then the voice recognition portion 54 converts the inputtedvoice into a text by a voice recognition process, and the tag settingportion 48 sets the text as a tag to the region of interest (S74). If noregion of interest is set as yet, then the voice recognition portion 54sets, for example, an object located at the center of the picked upimage or an object which occupies a greater area than any other objectin the picked up image to the region of interest, and sets the tag tothe region of interest. If there is no voice input (N at S72), then theprocessing advances to step S76.

The posture detection unit 60 acquires posture information of theportable terminal 200, namely, a tilt angle, a roll angle and a panangle (S76). The component image generation unit 20 stores the postureinformation of the portable terminal 200 and the region-of-interestinformation set by the POI/ROI setting unit 42 in an associatedrelationship with the picked up image as a component image into thecomponent image storage unit 22 (S78).

FIGS. 26(a) to 26(c) are views illustrating an example of a format of apicked up image to which a region of interest is set.

The information of a point of interest or a region of interest of arectangle is recorded into a geographical information tag data filecalled KML (Keyhole Markup Language) file depicted as an example in FIG.26(a). Reference numeral 450 describes information of a pan angle, atilt angle and a roll angle of a camera. Reference numeral 452 describesan image size. Reference numeral 454 describes whether the direction ofthe image is that of a landscape format or that of a portrait format.

Reference numeral 456 describes information for designating arectangular region of interest and designates coordinates of the centralpoint and an image size of the region of interest. Here, coordinates(200, 250) of the center of the region of interest and a width of 21 anda height of 15 of the image are designated. FIG. 26(b) depicts theregion of interest described by the reference numeral 456.

When a point of interest is designated, coordinates of a point ofinterest are designated and the image size is designated as zero asindicated by reference numeral 458. Here, coordinates (100, 150) of apoint of interest and a width of 0 and a height of 0 of the image aredesignated. FIG. 26(c) depicts a point of interest described byreference numeral 458.

Reference numeral 460 describes latitude and longitude informationobtained from the GPS receiver 74. Reference numeral 462 describes imagepickup date and time.

If the region of interest is not a rectangular region but a free region,then since the region of interest cannot be designated by coordinates, amask image is set to an α plane or a layer for exclusive use to beoverlapped with a panoramic image.

FIGS. 27(a) to 27(f) are views illustrating methods in which a user setsa region of interest on the touch panel.

Since, in panoramic image pickup, an image of a landscape over a widerange is picked up as depicted in FIG. 27(a), it is not easy torecognize from a synthesized panoramic image in which one of subjectsthe user is interested. Therefore, the user is caused to designate aregion of interest on the touch panel in a picked up image displayed onthe display 240 during panoramic image pickup depicted in FIG. 27(b).

FIG. 27(c) illustrates an example wherein an object in a picked up imageon the touch panel is set as a region of interest by tapping ordouble-tapping (reference numeral 430) the object. At this time, aregion including the object is automatically extracted from within thepicked up image and set as a region of interest.

FIG. 27(d) illustrates an example wherein an object is surrounded asdenoted by reference numeral 432 to designate a region of interest. Apolygon 434 proximate to a free curve denoted by reference numeral 432is set as a region of interest or a rectangular region 436circumscribing the polygon 434 is set as a region of interest.

FIG. 27(e) illustrates an example wherein a region of interest is set byplacing an oblique line into an object as denoted by reference numeral438. A rectangular region 440 having a diagonal line as the oblique lineis set as a region of interest.

FIG. 27(f) illustrates an example wherein a region of interest is set bypinching an object with two fingers as denoted by reference symbols 442a and 442 b. A rectangular region 444 having a diagonal line coincidentwith a line segment interconnecting start points of the pinch of the twofingers is set as a region of interest.

FIG. 28 is a view illustrating a region of interest set to an α plane ora layer for exclusive use to be overlapped with a panoramic image. Whena region of interest is designated by a free curve denoted by referencenumeral 470 on a picked up image, the region of interest is set as amask region 472 on the α plane or the layer for exclusive use denoted byreference numeral 474. A degree of interest can be recorded with binaryvalues such that it is 1 in a masked region and it is 0 in a non-maskedregion. Further, in the case of the α plane, also it is possible torecord a degree of interest in multiple values depending upon the αvalue of the masked region.

With the panoramic image generation apparatus of the present embodiment,a region of interest can be set during panoramic image pickup. Differentfrom ordinary image pickup wherein a subject of interest is focused torelease the shutter, in panoramic image pickup, an image of subjectsover a wide range is picked up, and therefore, it is frequently unknownin which one of subjects the user is interested to carry out imagepickup. In this regard, in the present embodiment, since a region ofinterest is designated during panoramic image pickup, a region in whichthe user is interested in a synthesized panoramic image can be grasped.Further, the labor for setting a region of interest in a panoramic imageafter synthesis can be omitted. Further, although, in the case of apanoramic image after synthesis, a region in which the user wants to beinterested cannot be displayed without scrolling on the screen image,during image pickup, it is possible to designate a region of interestsimply on the picked up image at present. Further, it is possible toautomatically set an object in which the user is interested in responseto the facial expression of the user during image pickup as a region ofinterest and associate the degree of interest or tag information byvoice with the region of interest.

Embodiment 4

FIG. 29 is a functional block diagram of a panoramic image generationapparatus 100 according to an embodiment 4. The panoramic imagegeneration apparatus 100 of the present embodiment has a function ofpicking up a panoramic image using both of a front side camera 210 and arear side camera 212 of a portable terminal 200. Those components whichare common to those of the panoramic image generation apparatus 100having the region-of-interest setting function of the embodiment 3described hereinabove with reference to FIG. 24 are denoted by likereference numerals and overlapping description of them is omittedherein.

In the present embodiment, as the mode for panoramic image pickup, thefollowing three modes are available.

(1) A “double-sided panoramic image pickup mode” in which a panoramicimage of one half circumference picked up by the front side camera 210and another panoramic image of one half circumference picked up by therear side camera 212 are connected to each other to pick up a panoramicimage of the whole circumference.

(2) A “panoramic image pickup mode with POI/ROI” in which a panoramicimage is picked up by the rear side camera 212 while the front sidecamera 210 is used to pick up an image of the face of the cameraoperator to set a point of interest or a region of interest.

(3) A “single-sided panoramic image pickup mode” in which a panoramicimage of the whole circumference is picked up by each of the front sidecamera 210 and the rear side camera 212. In this instance, two wholecircumference panoramic images are obtained. However, after the imagepickup, the user would determine whether the whole circumferencepanoramic image picked up by one of the cameras, specifically by thefront side camera 210 by which the camera operator is caught with a highdegree of possibility, is to be discarded or the two whole circumferencepanoramic images are to be utilized as a stereo panoramic image or aspanoramic images of a near view/distant view.

An image pickup mode changeover unit 94 carries out changeover to one ofthe above-mentioned panoramic image pickup modes. The user may manuallyset a panoramic image pickup mode on a setting screen image or the likesuch that the image pickup mode changeover unit 94 carries outchangeover to the image pickup mode set by the user. Alternatively, theimage pickup mode changeover unit 94 may automatically carry outchangeover to an appropriate panoramic image pickup mode in response toan image pickup situation of a panoramic image.

Alternatively, changeover among the image pickup modes may be carriedout in the following manner. In particular, at the point of time ofstarting of image pickup, a panoramic image is picked up by each of thefront side camera 210 and the rear side camera 212 withoutdistinguishing the double-sided panoramic image pickup mode of (1) andthe single-sided panoramic image pickup mode of (3). Then, at a point oftime at which image pickup for one half circumference is completed, inthe case in which the consistency on the image boundary between thefront side panoramic image of the one half circumference picked up bythe front side camera 210 and the rear side panoramic image of the onehalf circumference picked up by the rear side camera 212 is good, theimage pickup mode is determined to the double-sided panoramic imagepickup mode of (1), but in the case in which the consistency is notgood, the image pickup mode is determined to the single-sided panoramicimage pickup mode of (3). The case in which the consistency is not goodis a case in which the camera operator is caught unintentionally(without the object of setting a region of interest) in the front sidepanoramic image, another case in which a horizontal parallax between thefront side camera 210 and the rear side camera 212 gives rise to aparallax between the front side panoramic image and the rear sidepanoramic image, or a like case.

The image pickup mode changeover unit 94 notifies the image pickup unit10 and the panoramic image generation unit 30 of the panoramic imagemode set as described above. The image pickup unit 10 picks up an imageof a subject using the front side camera 210 and the rear side camera212 in accordance with the designated panoramic image pickup mode. Thepanoramic image generation unit 30 combines the component images pickedup by the image pickup unit 10 in accordance with the designatedpanoramic image pickup mode to generate a panoramic image.

When the image pickup mode is the double-sided panoramic image pickupmode, the panoramic image generation unit 30 connects the rear sidepanoramic image of the one half circumference obtained by combination ofthe rear side component images picked up by the rear side camera 212 andthe front side panoramic image of the one half circumference on theopposite side different by 180 degrees which image is obtained bycombination of the front side component images picked up by the frontside camera 210 to each other to generate a whole circumferencepanoramic image.

When the image pickup mode is the single-sided panoramic image pickupmode, the panoramic image generation unit 30 discards the front sidecomponent images of the whole circumference picked up by the front sidecamera 210 and combines the rear side component images of the wholecircumference picked up by the rear side camera 212 to generate a wholecircumference panoramic image. Although this is because the cameraoperator is frequently caught unintentionally in the picked up images bythe front side camera 210, depending upon an image pickup situation, therear side component images picked up by the rear side camera 212 may bediscarded while the front side component images picked up by the frontside camera 210 are combined to generate a whole circumference panoramicimage. Alternatively, the panoramic image generation unit 30 maysynthesize, without discarding the component images picked up by one ofthe cameras, whole circumference panoramic images from the componentimages picked up by both cameras and use the whole circumferencepanoramic images as a stereo panoramic image or as panoramic images of anear view/distant view.

After the portable terminal 200 is panned over one half circumference toallow each of the front side camera 210 and the rear side camera 212 topick up a panoramic image for one half circumference, when the frontside panoramic image picked up by the front side camera 210 and the rearside panoramic image picked up by the rear side camera 212 are connectedto each other to generate a panoramic image for the whole circumference,an image correlation decision unit 96 calculates a correlation betweenthe front side panoramic image and the rear side panoramic image in aregion in the proximity of the boundary of the connection and decideswhether or not the two images have a fixed consistency therebetween.

If the front side panoramic image and the rear side panoramic image havea correlation equal to or higher than a predetermined threshold value,then the image correlation decision unit 96 notifies the image pickupmode changeover unit 94 and the image pickup guide portion 52 that thefront side panoramic image and the rear side panoramic image have aconsistency therebetween. However, if the front side panoramic image andthe rear side panoramic image have a correlation lower than thepredetermined threshold value, then the image correlation decision unit96 notifies the image pickup mode changeover unit 94 and the imagepickup guide portion 52 that the front side panoramic image and the rearside panoramic image have no consistency therebetween.

If the front side panoramic image and the rear side panoramic image havea consistency therebetween, then the image pickup mode changeover unit94 sets the panoramic image pickup mode to the double-sided panoramicimage pickup mode. However, if the front side panoramic image and therear side panoramic image have no consistency therebetween, then theimage pickup mode changeover unit 94 sets the panoramic image pickupmode to the single-sided panoramic image pickup mode.

If the front side panoramic image and the rear side panoramic image havea consistency therebetween, then the image pickup guide portion 52 ofthe user interface unit 50 notifies the user that panoramic image pickupis completed by the panning over the one half circumference of theportable terminal 200. Further, when the front side panoramic image andthe rear side panoramic image have a consistency therebetween, the useris guided by a dialog or the like so that panning is carried out overanother one half circumference to carry out whole circumferencepanoramic image pickup.

The camera operator detection portion 90 of the image pickup unit 10detects whether or not the camera operator is caught in the imagespicked up by the front side camera 210 and notifies, if the cameraoperator is caught, a camera operator image pickup permission portion 57of the user interface unit 50 of the fact. The camera operator imagepickup permission portion 57 displays such a dialog as “Your face iscaught. OK?” and allows the user to issue an instruction regardingwhether or not the user permits image pickup of the face of the useritself by the front side camera 210. If the user permits image pickup ofthe face thereof, then the image pickup mode changeover unit 94 sets thepanoramic image pickup mode to the panoramic image pickup mode withPOI/ROI, and image pickup by the front side camera 210 is executeddirectly. If the user does not permit image pickup of the face thereof,then the images in which the camera operator is caught are discarded,and at a point of time at which the camera operator is not caught anymore, image pickup by the front side camera 210 is resumed. The imagesin which the camera operator is caught by the front side camera 210 areused when the POI/ROI setting unit 42 sets a region of interest or setsa degree of interest as described hereinabove in the description of theembodiment 3.

FIGS. 30(a) to 30(c) are views illustrating disposition of and aparallax between the front side camera 210 and the rear side camera 212of the portable terminal 200.

If the front side camera 210 and the rear side camera 212 are disposedat positions different in a horizontal direction from each other asdepicted in FIG. 30(a), then both camera have a horizontal parallax(parallax) HP therebetween. If the front side camera 210 and the rearside camera 212 are disposed at positions different also in the verticaldirection from each other, then both cameras have a vertical parallax VPtherebetween. Here, for the convenience of description, it is assumedthat the vertical parallax VP is zero or so small that it can be ignoredwhile attention is paid to the horizontal parallax HP.

It is assumed that the front side camera 210 and the rear side camera212 individually have a horizontal view angle of 75 degrees as depictedin FIG. 30(b). Since the front side camera 210 and the rear side camera212 have the horizontal parallax HP therebetween, a displacement by theparallax occurs between a front side panoramic image picked up by thefront side camera 210 and a rear side panoramic image picked up by therear side camera 212. Specifically, when an image of a near view ispicked up, the displacement by the parallax which appears between thefront side panoramic image and the rear side panoramic image cannot beignored.

However, when an image of a distant view is picked up, the displacementby the parallax which appears between a front side panoramic imagepicked up by the front side camera 210 and a rear side panoramic imagepicked up by the rear side camera 212 can be ignored. FIG. 30(c) depictsdisposition of and an angle of view between the front side camera 210and the rear side camera 212 when the displacement by the parallax canbe ignored. When the displacement by the parallax can be ignored, thefront side camera 210 and the rear side camera 212 can be treated suchthat they are disposed at the same position in a horizontal direction asdepicted in FIG. 30(c). If it is assumed that the horizontal angle ofview of both cameras is 75 degrees, then if, from a result of thecalculation of 90 degrees−(75 degrees/2)=52.5 degrees, the portableterminal 200 is rotated by 52.5 degrees to the left and right asindicated by broken lines of FIG. 30(c), then the front side camera 210can pick up a front side panoramic image for 180 degrees of one halfcircumference while the rear side camera 212 can pick up a planarpanoramic image for 180 degrees of one half circumference. Thus, if thefront side panoramic image of the one half circumference and the rearside panoramic image of the one half circumference are connected to eachother, then a whole circumference panoramic image can be obtained. Withsuch a rotational angle as 52.5 degrees, the user can carry outpanoramic image pickup while watching the display 240 as a finder.

When an image of a near view is to be picked up, since the front sidepanoramic image by the front side camera 210 and the rear side panoramicimage by the rear side camera 212 have a displacement by a parallaxtherebetween, even if the front side panoramic image of one halfcircumference and the rear side panoramic image of the other halfcircumference are connected to each other, a sufficient consistencycannot be obtained and an accurate whole circumference panoramic imagemay not be able to be generated. Therefore, when an image of a near viewis to be picked up, rotation by a further one half circumference iscarried out such that a whole circumference panoramic image is picked upby each of the front side camera 210 and the rear side camera 212. Inthis case, since two whole circumference panoramic images having aparallax therebetween are obtained, they may be used as a stereopanoramic image to display a three-dimensional panoramic image. Further,since depth information is obtained, also it is possible to generate athree-dimensional mode of a subject.

In the case of near view image pickup, the image pickup guide portion 52may urge the user to carry out whole circumference image pickup so thateach of the front side camera 210 and the rear side camera 212 can pickup a whole circumference panoramic image. However, in the case ofdistant view image pickup, the image pickup guide portion 52 may notifythe user that, at a point of time at which each of the front side camera210 and the rear side camera 212 picks up a panoramic image for one halfcircumference, whole circumference panoramic image pickup is completed.The user may designate whether an image of a near view is to be pickedup or an image of a distant view is to be picked up. For example, theimage pickup may be automatically set such that, when the camera is setto wide angle side image pickup, it is set to near view image pickup,but when the camera is set to telephoto side image pickup, it is set todistant view image pickup.

FIG. 31(a) is a view illustrating cylindrical images 340 a to 340 fpicked up by the rear side camera 212. Although they are similar to theimages described hereinabove in connection with FIG. 8B, it is assumedhere that, if the images 340 a to 340 f are combined, then a rear sidepanoramic image for one half circumference is obtained as depicted inFIG. 32(a). FIG. 31(b) is a view illustrating cylindrical images 350 ato 350 f picked up by the front side camera 210. If the images 350 a to350 f are combined, then a front side panoramic image for one halfcircumference is obtained as depicted in FIG. 32(b).

FIG. 32(a) is a view illustrating a rear side panoramic image picked upby the rear side camera 212. FIG. 32(b) is a view illustrating a frontside panoramic image picked up by the front side camera 210. FIG. 33 isa view illustrating a state in which the rear side panoramic imagepicked up by the rear side camera 212 and the front side panoramic imagepicked up by the front side camera 210 are connected to each other. Inregions 370 a to 370 d in the proximity of boundaries of the connectionat which the rear side panoramic image and the front side panoramicimage overlap with each other, the image correlation decision unit 96calculates a correlation between the rear side panoramic image and thefront side panoramic image. When the correlation is high, namely, whenthe number of pixels having pixel values coincide with or near to eachother is great, in a region in which the two images overlap with eachother, the image correlation decision unit 96 decides that the twoimages have a fixed consistency. However, when the two images are low incorrelation, namely, when the number of pixels having pixel valuescoincident with or near to each other is small, in the overlappingregions thereof, the image correlation decision unit 96 decides that thetwo images have no consistency therebetween.

FIG. 34 is a view illustrating an example of a front side panoramicimage 382 picked up by the front side camera 210 and a rear sidepanoramic image 380 picked up by the rear side camera 212. The cameraoperator is caught large in the front side panoramic image 382, and in aregion 370 d in the proximity of a boundary along which the front sidepanoramic image 382 and the rear side panoramic image 380 are connectedto each other, a hand of the camera operator is caught. Therefore, thecorrelation between the two images is low. In this case, the imagecorrelation decision unit 96 decides that the consistency between thefront side panoramic image 382 and the rear side panoramic image 380 islow. Consequently, the image pickup mode changeover unit 94 changes overthe panoramic image pickup mode to the single-sided panoramic imagepickup mode in which panoramic image pickup is carried out only by therear side camera 212, and the image pickup guide portion 52 guides theuser so as to further pan the camera over one half circumference to pickup a rear side panoramic image of the whole circumference.

FIG. 35 is a view illustrating an example of a guide where theconsistency between the front side panoramic image 382 and the rear sidepanoramic image 380 is low. If the image correlation decision unit 96decides at a point of time at which the front side panoramic image 382and the rear side panoramic image 380 are each picked up over one halfcircumference that the front side panoramic image 382 and the rear sidepanoramic image 380 do not have a fixed consistency therebetween, thenthe image pickup guide portion 52 causes the screen to display a dialog386 “Please rotate over further one half circumference” and indicatesthe image pickup direction by an arrow mark 384. Consequently, when theuser picks up the rear side panoramic image 380 for one halfcircumference, the user does not stop the image pickup but can pan therear side camera 212 for further one half circumference thereby to pickup a rear side panoramic image for the whole periphery.

FIG. 36 is a flow chart illustrating a procedure of image pickup of apanoramic image by the panoramic image generation apparatus 100according to the embodiment 4.

The camera operator detection portion 90 detects whether or not thecamera operator itself is caught in an image picked up by the front sidecamera 210 (S80). If the camera operator is caught (Y at S80), then thecamera operator image pickup permission portion 57 asks the user aboutwhether or not an image of the user itself should be picked up bydisplaying a dialog or by outputting sound (S82). If the user permitsimage pickup of the user itself (Y at S82), then the image pickup modechangeover unit 94 sets the panoramic image pickup mode to the panoramicimage pickup mode with POI/ROI (S84) and then carries out panoramicimage pickup with POI/ROI (S86). The panoramic image pickup process withPOI/ROI at step S86 is similar to the image pickup procedure of apanoramic image of the embodiment 3 described hereinabove with referenceto FIG. 25. In particular, an image of the facial expression of the useris picked up by the front side camera 210, and a region of interest anda degree of interest are set based on the facial expression. Further,panoramic image pickup is carried out by the rear side camera 212 togenerate a panoramic image with POI/ROI.

If the user does not permit image pickup of the user itself at step S82(N at step S82), then the panoramic image pickup is interrupted whilethe camera operator remains caught in a picked up image by the frontside camera 210 (N at S88). After it is waited that the camera operatoris not caught any more (Y at S88), a panoramic image pickup process bythe double-sided cameras is carried out (S90).

Also when the camera operator is not detected in an image picked up bythe front side camera 210 at step S80 (N at step S80), a panoramic imagepickup process by the double-sided cameras is started (S90).

FIG. 37 is a flow chart illustrating a detailed procedure of thepanoramic image pickup process by the double-sided cameras at step S90of FIG. 36.

In the panoramic image pickup process by the double-sided cameras, at apoint of time of starting of image pickup, whether the panoramic imagepickup mode is the double-sided panoramic image pickup mode or thesingle-sided panoramic image pickup mode is not determined except a casein which a panoramic image pickup mode is set manually. To which one ofthe double-sided panoramic image pickup mode and the single-sidedpanoramic image pickup mode the panoramic image pickup mode is to bedetermined is not known until panoramic image pickup for one halfcircumference is completed and the consistency is decided between thefront side panoramic image for one half circumference and the rear sidepanoramic image for one half circumference.

A component image pickup process (S110) executes the image pickup timingdecision process (S10), image pickup (S12), posture informationacquisition process (S14) and process for associating postureinformation with a picked up image (S16) of the embodiment 1 describedhereinabove with reference to FIG. 10.

If the panoramic image pickup for one half circumference by the frontside camera 210 and the rear side camera 212 is not completed (N atS112), then the processing returns to step S110 to repeat the componentimage pickup process.

If the panoramic image pickup for one half circumference by the frontside camera 210 and the rear side camera 212 is completed (Y at S112),then the image correlation decision unit 96 decides a degree ofcorrelation between the front side component image for one halfcircumference picked up by the front side camera 210 and the rear sidecomponent image for one half circumference picked up by the rear sidecamera 212 in an overlapping region in the proximity of a boundary alongwhich the two images are connected to each other (S114). If the degreeof correlation of the overlapping region is equal to or higher than apredetermined threshold value (Y at S114), then the image pickup modechangeover unit 94 sets the panoramic image pickup mode to thedouble-sided panoramic image pickup mode (S117) and the panoramic imagegeneration unit 30 combines front side component images for one halfcircumference and rear side component images for one half circumferenceto generate a whole circumference panoramic image (S118). If the degreeof correlation of the overlapping region is lower than the predeterminedthreshold value (N at S114), then the image pickup mode changeover unit94 sets the panoramic image pickup mode to the single-sided panoramicimage pickup mode (S115) and the image pickup guide portion 52 guidesthe user so as to further carry out image pickup for one halfcircumference (S116). Thereafter, the processing advances to a componentimage pickup process at step S120.

The component image pickup process at step S120 is same as the componentimage pickup process at step S110. If the panoramic image pickup for thewhole circumference by the front side camera 210 and the rear sidecamera 212 is not completed (N at S122), then the processing returns tostep S110 to repeat the component image pickup process.

If the panoramic image pickup for the whole circumference by the frontside camera 210 and the rear side camera 212 is completed (Y at S122),then an inquiry about whether or not the front side component images forthe whole circumference are to be discarded is issued to the user(S124). For example, if the user is caught in the front side componentimages, then the front side component images are discarded. When thefront side component images for the whole circumference are to bediscarded (Y at S124), the panoramic image generation unit 30synthesizes a whole circumference panoramic image from the rear sidecomponent images for the whole circumference (S126).

If the front side component images for the whole circumference are notto be discarded (N at S124), then the panoramic image generation unit 30synthesizes a whole circumference front side panoramic image from thefront side component images for the whole circumference (S128) andsynthesizes a whole circumference rear side panoramic image from therear side component images for the whole circumference (S130). In thiscase, the whole circumference front side panoramic image and the wholecircumference rear side panoramic image are used to obtain a stereopanoramic image. Further, the whole circumference front side panoramicimage may be utilized as a panoramic image of a near view while thewhole circumference rear side panoramic image is utilized as a panoramicimage of a distant view. Alternatively, if the zoom ratios of the frontside camera 210 and the rear side camera 212 are made different fromeach other, then also it is possible to obtain two panoramic images ofdifferent zoom ratios.

With the panoramic image generation apparatus of the present embodiment,panoramic image pickup can be carried out using both of the front sidecamera 210 and the rear side camera 212 of the portable terminal 200.Where the mounting positions of the two cameras are displaced from eachother, a parallax appears between panoramic images picked up by thecameras, and therefore, a stereo panoramic image can be acquired. Alsoit is possible to pick up a near view panoramic image by the front sidecamera 210 and pick up a distant view panoramic image by the rear sidecamera 212.

Further, if the double-sided cameras are utilized, then even if theportable terminal 200 is not rotated over 360 degrees, a panoramic imagefor the whole circumference can be picked up only by rotating theportable terminal 200 by 90 degrees−θ/2 with respect to the horizontalangle θ of view of the camera. Therefore, it is facilitated to carry outwhole circumference panoramic image pickup simply while the display ofthe portable terminal 200 is viewed as a finder.

While, in the example described above, panoramic image pickup where thefront side camera 210 and the rear side camera 212 are incorporated inthe portable terminal 200 is described, three or more cameras may beincorporated in the portable terminal 200. Further, the positions of theoptical axes of a plurality of cameras may be different in thehorizontal or vertical direction, and the directions of the optical axesof the plural cameras may be different from each other. The differencebetween the directions of the optical axes is not limited to thatbetween the opposite directions different by 180 degrees from eachother. In the following, variations of the cameras built in the portableterminal 200 are exemplified.

FIG. 38 is a view illustrating a portable terminal 200 in which twocameras having optical axes whose positions and directions are differentfrom each other are incorporated. While, in FIG. 30(b), the positions ofthe optical axes of the front side camera 210 and the rear side camera212 are different from each other in the horizontal direction, the frontside camera 210 and the rear side camera 212 in FIG. 38 are differentnot only in the position of the optical axis but also in the directionof the optical axis, and the front side camera 210 is directedobliquely.

FIG. 39 is a view illustrating a portable terminal 200 in which threecameras having optical axes whose positions are different from oneanother are incorporated. Two rear side cameras 212 a and 212 b and onefront side camera 210 are incorporated in the portable terminal 200. Thepositions of the optical axes of the three cameras 212 a, 212 b and 210are different from one another. A stereo panoramic image can be pickedup by the two rear side cameras 212 a and 212 b.

FIG. 40 is a view illustrating a portable terminal 200 in which threecameras having optical axes whose orientations are different from oneanother are incorporated. The portable terminal 200 has such a shape as,for example, a cylindrical shape, and has a display 240 on the frontface thereof. When the portable terminal 200 is viewed from above, thethree cameras 211 a, 211 b and 211 c are disposed substantiallyuniformly on a circle. The orientations of the optical axes of the threecameras 211 a, 211 b and 211 c are different from one another. Apanoramic image of the whole circumference can be picked up by rotatingthe portable terminal 200.

FIGS. 41(a) and 41(b) are views illustrating portable terminals 200 inwhich two cameras having different performances from each other areincorporated. In FIG. 41(a), while the horizontal angle of view of thefront side camera 210 is 50 degrees, the horizontal angle of view of therear side camera 212 is 120 degrees. In this manner, a plurality ofcameras having different angles of view from each other may beincorporated in the portable terminal 200.

In FIG. 41(b), while the front side camera 210 has a resolution of VGAand a frame rate of 60 fps (frames per second), the rear side camera 212has a resolution of 12 mega pixels and a frame rate of 15 fps. While thefront side camera 210 has a low resolution, it has a high frame rate andis advantageous in image pickup of a moving body. While the rear sidecamera 212 has a high resolution, it has a low frame rate and isunsuitable for image pickup of a moving body. However, the rear sidecamera 212 is advantageous in image pickup of a stationary body. In thismanner, a plurality of cameras which are different in performance suchas a resolution or a frame rate may be incorporated in the portableterminal 200.

FIGS. 42(a) to 42(c) are views illustrating variations of the number andthe orientation of cameras mounted on the portable terminal 200. FIG.42(a) depicts an example wherein a camera 211 c is mounted on a frontface, another camera 211 a on a rear face, a further camera 211 d on aleft side face and a still further camera 211 b on a right side face ofthe portable terminal 200. FIG. 42(b) depicts an example wherein acamera 211 a is mounted on a rear face, another camera 211 c on a leftside face and a further camera 211 b on a right side face of theportable terminal 200. FIG. 42(c) depicts an example wherein a camera211 a is mounted on a rear face and another camera 211 b is mounted on aright side face of the portable terminal 200. While the description hereis given of examples of the portable terminal 200, the cameras may bevehicle-carried cameras. Especially, in the case of vehicle-carriedcameras, the orientations of the cameras can be set to the leftward andrightward orientations in addition to the forward and rearwardorientations.

FIG. 43 is a view illustrating a portable terminal 200 in which fourcameras are incorporated. A camera 211 c is mounted on a front face,another camera 211 a on a rear face, a further camera 211 d on a leftside face and a still further camera 211 b on a right side face of theportable terminal 200. Since the cameras have a sufficiently largehorizontal angle of view, even if the portable terminal 200 is notrotated in a panning direction, an omnidirectional panoramic image canbe picked up by the four cameras 211 a to 211 d. However, whereomnidirectional image pickup cannot be carried out in a tiltingdirection, a panoramic image for the celestial sphere can be picked upby rotating the portable terminal 200 in the tilting direction.

FIGS. 44(a) and 44(b) are views illustrating portable terminals 200 inwhich movable cameras whose orientation can be adjusted areincorporated. In FIG. 44(a), while a camera 211 a mounted on a rear faceof the portable terminal 200 is a fixed camera, another camera 211 bmounted on a right side face of the portable terminal 200 is a movablecamera and has an orientation which can be adjusted manually orautomatically. In FIG. 44(b), while a first camera 211 a mounted on arear face of the portable terminal 200 is a fixed camera, a secondcamera 211 b is a movable camera whose orientation can be reversedforwardly and rearwardly. Therefore, if the orientation of the camera211 b is reversed, then the camera 211 b can serve as a rear side cameraand also as a front side camera. The orientation of the movable cameracan be detected automatically by the portable terminal 200.

The variations of the camera incorporated in the portable terminal 200described above are absolutely illustrative, and various othercombinations of cameras may be available.

By incorporating a plurality of cameras which are different at least inone of the position and the orientation of the optical axis therebetweenor thereamong in this manner, images picked up using the cameras arecombined to synthesize an omnidirectional panoramic image.

The image pickup mode changeover unit 94 changes over the panoramicimage pickup mode between a compound eye panoramic image pickup mode inwhich one “compound eye panoramic image” formed by connecting panoramicimages picked by the plural cameras incorporated in the portableterminal 200 is picked up and a single eye panoramic image pickup modein which a “single eye panoramic image” is picked up by each of theplural cameras.

When the panoramic image pickup mode is the compound eye panoramic imagepickup mode, the panoramic image generation unit 30 combines a pluralityof component images picked up by the cameras to generate a single eyepanoramic image, and then connects the single eye panoramic images bythe cameras to each other to generate a compound eye panoramic image. Inthe case of the single eye panoramic image pickup mode, the panoramicimage generation unit 30 combines a plurality of component images pickedup by each camera to generate a single eye panoramic image and thencombines the single eye panoramic images by the cameras to generate andoutput a stereo panoramic image or a plurality of panoramic images whichare different from each other in the sense of distance such as those ofa near view and a distant view.

The image correlation decision unit 96 decides, in a boundary region inwhich single eye panoramic images by the cameras are connected to eachother, a correlation degree between the single eye panoramic images tobe connected to each other. When the correlation degree is equal to orhigher than a predetermined threshold value, the panoramic imagegeneration unit 30 connects the single eye panoramic images by thecameras to each other to generate a compound eye panoramic image.Although the plural cameras incorporated in the portable terminal 200cannot carry out omnidirectional panoramic image pickup where theyremain at the individually fixed positions, if they are rotated in a pandirection or a tilt direction, then the fields of view in which imagepickup of the cameras can be carried out come to overlap with eachother. By evaluating the correlation between the single eye panoramicimages to be connected to each other in the overlapping regions, it canbe decided whether the single eye panoramic images by the cameras may beconnected to each other by a stitching process, whether the single eyepanoramic images by the cameras should be outputted as separate imagesfrom each other or whether the images by one of the cameras should bediscarded.

The present invention has been described above in connection with theembodiments thereof. The embodiments are exemplary, and it is to berecognized by those skilled in the art that various modifications arepossible to the combinations of the components and/or the processes ofthe embodiments and that also such modifications are included in thescope of the present invention.

While the embodiments have been described taking a celestial spherepanoramic image as an example of a panoramic image, the panoramic imageneed not be a celestial sphere panoramic image but may be an imageformed by combining a plurality of images picked up by a camera whilethe image pickup direction is successively changed.

It is to be noted that the “panoramic image” in the presentspecification is not limited to a “panoramic image in a narrow sense,namely, an image of a landscape format or a portrait format, a full viewimage over 360 degrees and the like, but merely is an image whose targetis a wide range. Further, while, in the description of the embodiments,an example wherein a panoramic image is generated as a synthesis image,the synthesis image to be outputted need not be a so-called panoramicimage, but the present invention can be applied also where the synthesisimage is an ordinary image of an arbitrary size. Alternatively, thesynthetic image to be outputted may be an image wherein a plurality ofimages having different resolutions from each other are hierarchized.Such a hierarchized image may be configured such that, if some region ofthe image is expanded, then the expanded region is replaced by an imageof a higher resolution.

REFERENCE SIGNS LIST

10 Image pickup unit, 12 Frame memory, 14 Image pickup timing decisionunit, 20 Component image generation unit, 22 Component image storageunit, 24 Non-image pickup region decision unit, 30 Panoramic imagegeneration unit, 32 Panoramic image storage unit, 40 Display unit, 42POI/ROI setting unit, 44 Region-of-interest setting portion 44, 46Degree-of-interest setting portion 46, 48 Tag setting portion 48, 50User interface unit, 52 Image pickup guide portion, 54 Voice recognitionportion, 56 Touch panel inputting portion, 57 Camera operator imagepickup permission portion 57, 58 POI/ROI processing unit, 60 Posturedetection unit, 62 Three-axis gyro sensor, 64 Three-axis accelerationsensor, 66 Orientation detection unit, 68 Three-axis geomagnetic sensor,70 Pan angle-elevation angle calculation unit, 72 Position detectionunit, 74 GPS receiver, 80 Communication unit, 82 Characteristic pointextraction unit, 84 Self-position estimation unit, 86 Camera positiondisplacement decision unit, 90 Camera operator detection portion, 92Facial expression decision portion, 94 Image pickup mode changeoverunit, 96 Image correlation decision unit, 100 Panoramic image generationapparatus, 200 Portable terminal, 210 Front side camera, 212 Rear sidecamera, 240 Display unit.

INDUSTRIAL APPLICABILITY

The present invention can be utilized for a technology for generating asynthesis image.

The invention claimed is:
 1. An image generation apparatus, comprising:a characteristic point extraction circuit configured to extract aplurality of characteristic points in an overlapping region of aplurality of component images, the plurality component images obtainedby capturing a plurality of respective sequential images using a camerabuilt in portable equipment, where each of the plurality of componentimages have a different pan angle or elevation angle relating to therespective sequential image, and a respective posture information beinga three-axis inclination of the portable equipment relating to therespective sequential image; a self-position estimation circuitconfigured to estimate, for each of two sequential component images inwhich camera positions are displaced from one another, a plurality ofrespective three-dimensional coordinate values of the respectivecharacteristic points in the overlapping region and respectivethree-dimensional coordinate value of the camera position; a componentimage generation circuit configured to associate, for each of the twosequential component images, the posture information, the plurality ofrespective three-dimensional coordinate values of the respectivecharacteristic points and the three-dimensional coordinate values of thecamera positions; a synthesis image generation circuit configured toadjust, based on the estimated three-dimensional coordinate values ofthe camera positions, the plurality of respective three-dimensionalcoordinate values of the respective characteristic points of thecomponent images to correct the component images and to combine the twocomponent images to generate a panoramic image; and an image pickupguide portion configured to display a locus of movement of the cameraposition based on the respective three-dimensional coordinate values ofthe camera position estimated by the self-position estimation circuit.2. The image generation apparatus according to claim 1, furthercomprising a camera position displacement decision circuit configured todecide whether or not the displacement of the camera position is withina permissible range based on the respective three-dimensional coordinatevalues of the camera position estimated by the self-position estimationcircuit; when the displacement of the camera position exceeds thepermissible range, the image pickup guide portion instructing a user tocorrect the camera position.
 3. The image generation apparatus accordingto claim 2, wherein the camera position displacement decision circuitsets a threshold value for deciding whether or not the displacement ofthe camera position is within the permissible range and sets thethreshold value such that the threshold value has a higher value as adistance to a subject determined from the plurality of respectivethree-dimensional coordinate values of the respective characteristicpoints estimated by the self-position estimation circuit and therespective three-dimensional coordinate values of the camera positionincreases.
 4. The image generation apparatus according to claim 3,wherein the camera position displacement decision circuit sets, as thethreshold value, a first threshold value and a second threshold valuehigher than the first threshold value; and the image pickup guideportion: (a) does not instruct, when it is decided that the displacementof the camera position is lower than the first threshold value, the userto correct the camera position but permits pickup of a component imageat the camera position, (b) instructs, when it is decided that thedisplacement of the camera position is equal to or higher than the firstthreshold value but is lower than the second threshold value, the userto correct the camera position and permits pickup of a component imageat the camera position, and (c) instructs, when the displacement of thecamera position is equal to or higher than the second threshold value,the user to correct the camera position but does not permit pickup of acomponent image at the camera position until the camera position becomesat least lower than the second threshold value.
 5. The image generationapparatus according to claim 4, wherein the first threshold value is anupper limit to a permissible displacement amount for the camera positionwith which the component images can be combined to generate a synthesisimage even if the component images are not corrected, and the secondthreshold value is an upper limit to a permissible displacement amountfor the camera position with which, if the component images arecorrected, then the component images after the correction can becombined to generate a synthesis image; and the synthesis imagegeneration circuit corrects and combines, when it is decided by thecamera position displacement decision circuit that the displacement ofthe camera position is equal to or higher than the first threshold valuebut is lower than the second threshold value, the component images togenerate a synthesis image.
 6. An image generation method, comprising:extracting a plurality of characteristic points in an overlapping regionof a plurality of component images, the plurality component imagesobtained by capturing a plurality of respective sequential images usinga camera built in portable equipment, where each of the plurality ofcomponent images have a different pan angle or elevation angle relatingto the respective sequential image, and a respective posture informationbeing a three-axis inclination of the portable equipment relating to therespective sequential image; estimating, for each of two sequentialcomponent images in which camera positions are displaced from oneanother, a plurality of respective three-dimensional coordinate valuesof the respective characteristic points in the overlapping region andrespective three-dimensional coordinate value of the camera position;associating, for each of the two sequential component images, theposture information, the plurality of respective three-dimensionalcoordinate values of the respective characteristic points and thethree-dimensional coordinate values of the camera positions; adjusting,based on the estimated three-dimensional coordinate values of the camerapositions, the plurality of respective three-dimensional coordinatevalues of the respective characteristic points of the component imagesto correct the component images and to combine the two component imagesto generate a panoramic image; and displaying a locus of movement of thecamera position based on the respective three-dimensional coordinatevalues of the camera position estimated by the self-position estimationcircuit.
 7. A non-transitory, computer-readable recording medium storinga program, the program causing a computer to carry out actions,comprising: extracting a plurality of characteristic points in anoverlapping region of a plurality of component images, the pluralitycomponent images obtained by capturing a plurality of respectivesequential images using a camera built in portable equipment, where eachof the plurality of component images have a different pan angle orelevation angle relating to the respective sequential image, and arespective posture information being a three-axis inclination of theportable equipment relating to the respective sequential image;estimating, for each of two sequential component images in which camerapositions are displaced from one another, a plurality of respectivethree-dimensional coordinate values of the respective characteristicpoints in the overlapping region and respective three-dimensionalcoordinate value of the camera position; associating, for each of thetwo sequential component images, the posture information, the pluralityof respective three-dimensional coordinate values of the respectivecharacteristic points and the three-dimensional coordinate values of thecamera positions; adjusting, based on the estimated three-dimensionalcoordinate values of the camera positions, the plurality of respectivethree-dimensional coordinate values of the respective characteristicpoints of the component images to correct the component images and tocombine the two component images to generate a panoramic image; anddisplaying a locus of movement of the camera position based on therespective three-dimensional coordinate values of the camera positionestimated by the self-position estimation circuit.